Loading, compiling and running a model instantiating PowerSysPro components are very time consuming

[jean-philippe.tavella@…]

I now try to instantiate PowerSysPro classes in a Modelica model.
This model can perfectly and efficiently be loaded, checked, compiled and run under Dymola.

With OM, loading, compiling and running are very very long process (very time consuming).
On the other hand, the check is very quick.

The number of equations / uncknowns seems small enough (less than 19,500).
Is there any explanation?

[Francesco Casella]

The check process is carried out by the new front end, which is extremely efficient, since it has been redesigned from scratch based on 10+ years of experience.

The compilation requires many other steps, some of which may scale badly, see #4146. We are currently in the process of also redesigning the backend from scratch, to be more efficient with large models, but that will take another 1-2 years.

To see which methods currently take most of the time, you need to turn on the -d=execstat flag, or to add ,execstat to existing -d debug flags if there are others active already. In OMEdit, you can do this in Simulation Setup | Translation Flags | Additional Translation Flags.

Then, you can see the time spent by each method in the whole code generation process, as well as the cumulated time spent from the beginning, so it's easy to spot if there are some that take a disproportionate amount of time.

[Francesco Casella]

Before getting there, I see that loading the model in OMEdit already takes a long time. I also noticed it takes a lot of memory (over 8 MB), which may explain why it is so slow on your computer, if memory swapping on disk starts because of lack of memory, the performance degrades dramatically.

@adeas31, @adrpo, why should that happen? This is quite fishy, I never noticed such an extreme behaviour upon loading a model in OMEdit.

[Francesco Casella]

I just tried to load the model, after 10 minutes and over 10 GB of memory allocation OMEdit crashed. I sent a crash report on my name.

I guess this kind of things shouldn't really happen.

[jean-philippe.tavella@…]

Replying to casella:

Before getting there, I see that loading the model in OMEdit already takes a long time. I also noticed it takes a lot of memory (over 8 MB), which may explain why it is so slow on your computer, if memory swapping on disk starts because of lack of memory, the performance degrades dramatically.

@adeas31, @adrpo, why should that happen? This is quite fishy, I never noticed such an extreme behaviour upon loading a model in OMEdit.

Good point for the check improvement and I can testify (at leat fo this model) it is now as quick as Dymola's one.

OK also for the compiler as we have a date goal around 2 years. Nevertheless I will test the flag as soon as possible.

My machine has 16 MB memory so it should be enough. Memory swapping should not occur.

[jean-philippe.tavella@…]

Replying to casella:

I just tried to load the model, after 10 minutes and over 10 GB of memory allocation OMEdit crashed. I sent a crash report on my name.

I guess this kind of things shouldn't really happen.

So much memory allocated for this (relatively small) model is not understandable.
And even with memory swapping crash should not occur.
Thank you for the priority blocker. I think effcient model loading should be done first.

[jean-philippe.tavella@…]

I tested the model loading on another machine with 32 GB memory.
I see loading takes about 3 minutes, and memory occupancy is slowly growing to reach 24.1 / 31.8 GB memory.

The same load with Dymola is immediate. The same when switching from the graphical view to the text view or from the text view to the graphical view.
Memory occupancy with Dymola is almost negligeable.

My model is automatically generated by some preprocessing tools we use at EDF, without annotation (no graphical information). It only contains class instantiations and a list of connect() to connect these instances (about 6,000 lines).

Maybe when loading in OMEdit, the default view could be the text view instead of the graphical view (if it can help to load it quicker)...

[Francesco Casella]

Replying to jean-philippe.tavella@…:

Good point for the check improvement and I can testify (at leat fo this model) it is now as quick as Dymola's one.

@perost and @adrpo did a very good job there :)

OK also for the compiler as we have a date goal around 2 years. Nevertheless I will test the flag as soon as possible.

If you notice that a major part of the time is spent on one or two methods, you can open a ticket about that and we'll look into it. Within 2 years you may actually get some benefits frome the new backend, currently under development, but if there is something that grossly stand out today, we can try to get it fixed in a much shorter time frame.

My machine has 16 MB memory so it should be enough. Memory swapping should not occur.

Absolutely.

[Adrian Pop]

First, I really hope when you write MB you talk about GB as I haven't seen a system with 16MB of RAM in a long time.

I will look into this but I need more info:

• what model you are loading?
• what library, where is it, what version or git hash?

I assume the repository is this one (as I added it to the library testing):
​https://bitbucket.org/simulage/powersyspro.git

[Adrian Pop]

Nevermind, I just saw the attachment. I'll start debugging this a bit.

[jean-philippe.tavella@…]

Replying to adrpo:

First, I really hope when you write MB you talk about GB as I haven't seen a system with 16MB of RAM in a long time.

I will look into this but I need more info:

• what model you are loading?
• what library, where is it, what version or git hash?

I assume the repository is this one (as I added it to the library testing):
​https://bitbucket.org/simulage/powersyspro.git

Yes, I'm very sorry the machine has 32 GB and memory occupancy when loading the model is slowly growing to reach 24.1 / 31.8 GB memory.
Yes the repository is correct for the library. You can get the 2.0.0 release directly available at ​https://bitbucket.org/simulage/powersyspro/src/master/ (PowerSysPro.mo) under the condition you use MSL 4.0.0.
The model to test instantiates classes from this library: the file has already been attached to this ticket.

[Adrian Pop]

As far as I can tell OMC is not to blame this time, it gives all the information to OMEdit and then OMEdit grows the memory *A LOT*. I attached with GDB to OMEdit and this is the trace:

(gdb) 
#0  0x00007ffcb98650ea in QGraphicsSceneInsertItemBspTreeVisitor::visit(QList<QGraphicsItem*>*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#1  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#2  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#3  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#4  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#5  0x00007ffcb97e0699 in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#6  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#7  0x00007ffcb97e0699 in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#8  0x00007ffcb97e06ee in QGraphicsSceneBspTree::climbTree(QGraphicsSceneBspTreeVisitor*, QRectF const&, int) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#9  0x00007ffcb97e1aee in QGraphicsSceneBspTreeIndexPrivate::_q_updateIndex() [clone .part.0] ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#10 0x00007ffcb97e2b64 in QGraphicsSceneBspTreeIndexPrivate::_q_updateSortCache() () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#11 0x00007ffcb97e37c7 in QGraphicsSceneBspTreeIndexPrivate::estimateItems(QRectF const&, Qt::SortOrder, bool) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#12 0x00007ffcb97e3aea in QGraphicsSceneBspTreeIndex::estimateTopLevelItems(QRectF const&, Qt::SortOrder) const ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#13 0x00007ffcb97dad2a in QGraphicsScenePrivate::drawItems(QPainter*, QTransform const*, QRegion*, QWidget*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#14 0x00007ffcb97f9db8 in QGraphicsView::paintEvent(QPaintEvent*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#15 0x00007ffcb95073b8 in QWidget::event(QEvent*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#16 0x00007ffcb95ae6d4 in QFrame::event(QEvent*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#17 0x00007ffcd72e2fac in QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*) [clone .part.0] ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#18 0x00007ffcb94c7fc2 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#19 0x00007ffcd72e47a0 in QCoreApplication::sendSpontaneousEvent(QObject*, QEvent*) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#20 0x00007ffcb94ffefb in QWidgetPrivate::sendPaintEvent(QRegion const&) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#21 0x00007ffcb950032f in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#22 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#23 0x00007ffcb9501ac8 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#24 0x00007ffcb9501ac8 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#25 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#26 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#27 0x00007ffcb9501ac8 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#28 0x00007ffcb9501ac8 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#29 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#30 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#31 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#32 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#33 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#34 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#35 0x00007ffcb9501ac8 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#36 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#37 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#38 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#39 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#40 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#41 0x00007ffcb9501ca6 in QWidgetPrivate::paintSiblingsRecursive(QPaintDevice*, QList<QObject*> const&, int, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#42 0x00007ffcb9500830 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#43 0x00007ffcb94d7d1a in QWidgetRepaintManager::paintAndFlush() ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#44 0x00007ffcb94d8728 in QWidgetRepaintManager::sync(QWidget*, QRegion const&) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#45 0x00007ffcb9522bf3 in QWidgetWindow::event(QEvent*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#46 0x00007ffcb94c7fd3 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Widgets.dll
#47 0x00007ffcd72e47a0 in QCoreApplication::sendSpontaneousEvent(QObject*, QEvent*) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#48 0x00007ffcb8d21519 in QGuiApplicationPrivate::processExposeEvent(QWindowSystemInterfacePrivate::ExposeEvent*) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Gui.dll
#49 0x00007ffcb8cf552c in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Gui.dll
#50 0x00007ffcb8cf5c48 in QWindowSystemInterface::flushWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Gui.dll
#51 0x00007ffcf5ada674 in QWindowsWindow::handleWmPaint(HWND__*, unsigned int, unsigned long long, long long) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\platforms\qwindows.dll
#52 0x00007ffcf5b7e8b4 in QWindowsContext::windowsProc(HWND__*, unsigned int, QtWindows::WindowsEventType, unsigned long long, long long, long long*, QWindowsWindow**) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\platforms\qwindows.dll
#53 0x00007ffcf5ae64f3 in qWindowsWndProc ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\platforms\qwindows.dll
#54 0x00007ffcfd9fe858 in USER32!CallWindowProcW ()
   from C:\WINDOWS\System32\user32.dll
#55 0x00007ffcfd9fe3dc in USER32!DispatchMessageW ()
   from C:\WINDOWS\System32\user32.dll
#56 0x00007ffcfda10bc3 in USER32!SendMessageTimeoutW ()
   from C:\WINDOWS\System32\user32.dll
#57 0x00007ffcfeccfbd4 in ntdll!KiUserCallbackDispatcher ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#58 0x00007ffcfc441704 in win32u!NtUserDispatchMessage ()
   from C:\WINDOWS\System32\win32u.dll
#59 0x00007ffcfd9fe2ea in USER32!DispatchMessageW ()
   from C:\WINDOWS\System32\user32.dll
#60 0x00007ffcd734020b in QEventDispatcherWin32::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#61 0x00007ffcf5b51db5 in QWindowsGuiEventDispatcher::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\platforms\qwindows.dll
#62 0x00007ffcd72e2843 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#63 0x00007ffcd72eb4f7 in QCoreApplication::exec() ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\Qt5Core.dll
#64 0x00007ff7b2c2189c in qMain (argc=2, argv=argv@entry=0x1d3a7f89ab0)
    at main.cpp:175
#65 0x00007ff7b3222914 in WinMain () at qtmain_win.cpp:97
#66 0x000000014066b542 in ?? ()
#67 0x00007ff7b2c213c1 in __tmainCRTStartup ()
    at C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:333
#68 0x00007ff7b2c214d6 in WinMainCRTStartup ()
    at C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:186

Maybe @adeas31 can find out what is going on.
It seems to be that OMEdit has issues handling a model with so many connections, there are 2996 of them. Truth is there are no graphical annotations but OMEdit needs to go through all the components and all the connections to build its structures.

[Francesco Casella]

@adrpo, @adeas, I did some experiments. In particular, I cut off the entire equation section of the test case, and also cut out 2/3 of the component declarations, keeping the first 1000 lines.

If you load that stripped model and try to open it in OMEdit, the API calls take a few seconds, then OMEdit goes on for a while, allocating several GB of RAM. Eventually you get the code to display, but there is clearly something fishy in how the information retrieved from the API is managed internally. I guess something grows as O(N²), or worse, N being the number of instantiated objects, my feeling is that as you grow the number of instantiated objects left in the code, things get worse in a more-than-linear fashion, but I had no patience to test that out.

At least we know the problem is with component instances, not connect statements.

BTW, this model has no graphical annotations on any component instance, so I really don't get the point of doing anything with that information at all. We should just display the code. Do I miss something?

[Christoph Buchner <buchner@…>]

@casella: Is a similar/equivalent case already present in the ScalableTestSuite? If not, it sounds like this problem case would be a good fit, no?

[Francesco Casella]

Most ScalableTestSuite models use arrays, not thousands of separate instances.

The only ones that are built like this example are the ones with Individual in their name, and indeed OMEdit gets stuck there. I actually never tried that before.

@adeas31, please also try opening those models, they can be found in the ScalableTestSuite.Electrical.DistributionSystemDC.ScaledExperiments package.

[Adeel Asghar]

1ddf691/OpenModelica is just a work around for this problem.

I will investigate more and will try to further improve the experience.

[Francesco Casella]

Great! Can you quickly explain what the problem is?

[Adeel Asghar]

I don't have an exact answer for it now.

What I have done is that I still create all the components but I just don't add them to the scene based on the annotation. What we were doing previously is that we are creating all the components and then if there is no annotation we call setVisible(false) on the component. So basically Qt has nothing to render, as per ​Qt docs the invisible items don't receive any events. But still somehow Qt rendering is going in some loop and allocating huge amounts of memory.

[Francesco Casella]

It looks like Qt gets into serious trouble with more than a few hundred objects. Is there any documentation available about that?

In any case, what's the point of wasting resources to create component that are not visible? Shouldn't we skip them anyway?

[jean-philippe.tavella@…]

Replying to casella:

It looks like Qt gets into serious trouble with more than a few hundred objects. Is there any documentation available about that?

In any case, what's the point of wasting resources to create component that are not visible? Shouldn't we skip them anyway?

Just a remark here: in the given model there are no annotations at all but we may have some with a mix of visible and not visible components.

[Adrian Pop]

There is another point, a question that showed up recently on some forum or Discord channel, I don't really remember (maybe @sjoelund.se does). A user asked why he can't see anything in the diagram view and then we told him this is because there are no graphical annotations in the model.

But currently we do have a all the information in OMEdit to display the model components (as boxes) and their connections and we could even do automatic layout using for example some Graphviz library. Then one could actually edit the parameters using the usual dialogs. Of course this needs to be activated via a setting but I think it would be a good feature to have.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

Just a remark here: in the given model there are no annotations at all but we may have some with a mix of visible and not visible components.

Sure. In that case, only the visible ones should probably be processed by QT.

[Francesco Casella]

Replying to adrpo:

But currently we do have a all the information in OMEdit to display the model components (as boxes) and their connections and we could even do automatic layout using for example some Graphviz library. Then one could actually edit the parameters using the usual dialogs. Of course this needs to be activated via a setting but I think it would be a good feature to have.

This would of course be a nice feature to have. Do you think it could scale up to 3000 components, as in this case?

[Adrian Pop]

Maybe not for so many components. But my feeling is that it should work even with so many. Seems to be a bug in qt with current behavior.

[jean-philippe.tavella@…]

Replying to adrpo:

There is another point, a question that showed up recently on some forum or Discord channel, I don't really remember (maybe @sjoelund.se does). A user asked why he can't see anything in the diagram view and then we told him this is because there are no graphical annotations in the model.

But currently we do have a all the information in OMEdit to display the model components (as boxes) and their connections and we could even do automatic layout using for example some Graphviz library. Then one could actually edit the parameters using the usual dialogs. Of course this needs to be activated via a setting but I think it would be a good feature to have.

This could be smart to have this default automatic layout as Dymola doesn't do that (as its diagram view is empty). But this is an enhencement and priority should be given to solve the Qt trouble.
We could also imagine that by default the text layer is used, not the diagram layer for models above a certain size. And if the user wants to swith from text to diagram, the tool could display a warning to say 'Hi, this could take a long time. do you confirm this action?"

[Francesco Casella]

Replying to jean-philippe.tavella@…:

This could be smart to have this default automatic layout as Dymola doesn't do that (as its diagram view is empty). But this is an enhencement and priority should be given to solve the Qt trouble.
We could also imagine that by default the text layer is used, not the diagram layer for models above a certain size. And if the user wants to swith from text to diagram, the tool could display a warning to say 'Hi, this could take a long time. do you confirm this action?"

That's exactly what I wanted to propose :)

[jean-philippe.tavella@…]

@casella and @adrpo: the attached model PowerGridElec.mo is one of those recently sent/shared by EDF in the context of the project EMBrACE.
For us, this model is an example of medium size model (not small, but not large too).

With Dymola,the model is:

• instantaneously loaded
• checked in 5 s
• compiled in 20 s
• run (with stop time = 0) in 2.2 s

With 4 FMUs exported from Dymola and co-simulated with DACCOSIM NG (on a >4-core computer), the case is run in 230 ms.

[Francesco Casella]

@jean-philippe, I'm analyzing your test.

In the meantime, please fix lines 226 and 574 of PowerSysPro.mo, you should write if Imax > 0 and if Smax > 0, since == and <> among Reals are forbidden in Modelica, see ​Modelica Specification, Section 3.5. I don't really know why Dymola accepts it. If you want to use Modelica code across multiple tools, it is very important to comply to the specification strictly.

I would have opened a pull request on bitbucket, but I'm used to github and I'm not sure how it works.

[jean-philippe.tavella@…]

Replying to casella:

@jean-philippe, I'm analyzing your test.

In the meantime, please fix lines 226 and 574 of PowerSysPro.mo, you should write if Imax > 0 and if Smax > 0, since == and <> among Reals are forbidden in Modelica, see ​Modelica Specification, Section 3.5. I don't really know why Dymola accepts it. If you want to use Modelica code across multiple tools, it is very important to comply to the specification strictly.

I would have opened a pull request on bitbucket, but I'm used to github and I'm not sure how it works.

I perfeclty agree (I thought it was already done, but apparently only for ==).
I have just uploaded the release 2.1.0 (Jan. 25th 2021) with these updates:

• Replacing in 3 models the <> by > as <> among Reals is forbidden in Modelica.
• Removing the model myVariablePVNode and modifying the examples related to breakers (islanding).

[Francesco Casella]

Replying to jean-philippe.tavella@…:

@casella and @adrpo: the attached model PowerGridElec.mo is one of those recently sent/shared by EDF in the context of the project EMBrACE.
For us, this model is an example of medium size model (not small, but not large too).

With Dymola,the model is:

• instantaneously loaded
• checked in 5 s
• compiled in 20 s
• run (with stop time = 0) in 2.2 s

I made a first run of performance evaluation on my PC, it is a i7-8550 Intel CPU running @ 1.8 GHz on Windows. If you have a 3 GHz workstation on Linux, speed may be twice or three times faster.

Once I have loaded the Modelica library 4.0.0 and the PowerSysPro library, the model file is loaded instantaneously. When you select it in the package browser, it takes about 6 s on my pc before it is visualized. This is because OMEdit calls the API on each and every component to fetch modifier values and graphical annotations. This would be needed if the model had graphical information (which it doesn't). There are about 12.000 such calls, even at 0.5 ms each this takes some time.

@adrpo, @adeas31 I'm not sure if we should try to make this faster, or rather if we should load and display the textual layer immediately, and use a parallel, non-blocking thread to fetch the information for the graphical view. I'll open a ticket about that later.

Checking takes 3 s.

C code generation takes 150 s, of which 80 are spent tearing the initial equations, and 20 are spent tearing the regular equations. Maybe we should just skip tearing, if you use sparse solvers you should get efficient simulation anyway. That would get code generation down to 50 s.

C compilation takes about 2 minutes. If you use a Linux system, clang is even faster than in Windows.

The simulation then starts and fails during initialization because of division by zero. This is due to insufficiently strict tearing rules, which lead to division by zero in the torn section. This is a known issue, see #6196, we should get this fixed in 1.17.0.

[Francesco Casella]

During compilation, I see the following warning:

[60] 22:09:36 Translation Warning
Assuming fixed start value for the following 720 variables:
         MVBB3_F6_BATIMENT0000000322361188.degradedMode:DISCRETE(start = false fixed = true protected = true )  "Mode for load: degraded / normal"PowerGridElec, PowerSysPro.Components.myVariableLoad type: Boolean
         MVBB3_F5_BATIMENT0000000356487694.degradedMode:DISCRETE(start = false fixed = true protected = true )  "Mode for load: degraded / normal"PowerGridElec, PowerSysPro.Components.myVariableLoad type: Boolean
         MVBB3_F5_BATIMENT0000000322361121.degradedMode:DISCRETE(start = false fixed = true protected = true )  "Mode for load: degraded / normal"PowerGridElec, PowerSysPro.Components.myVariableLoad type: Boolean
         MVBB3_F5_BATIMENT0000000322361232.degradedMode:DISCRETE(start = false fixed = true protected = true )  "Mode for load: degraded / normal"PowerGridElec, PowerSysPro.Components.myVariableLoad type: Boolean
...

These states are not explicitly set in initial equation, nor explicitly given a fixed = true. This requires OpenModelica to figure out what are the missing initial equations, which could lead to a different outcome than Dymola.

If you want to use the same Modelica model across tools, it is recommended to completely specify the initial conditions with initial equations and/or fixed = true attributes for all continuous and discrete states of your system.

[Francesco Casella]

OK, I was able to get your model to run by setting some flags:

• compiler flag -d=tearingMethod=minimalTearing avoids tearing, which speeds up code generation and avoids division by zero during initialization
• simulation flag -nls=kinsol uses the sparse KINSOL solver to efficiently solve the large nonlinear implicit system

I also set compiler flags -d=stateselection,execstat to get some more info about the model structure and about where the time is spent.

You can find the updated model with the extra annotations attached.

Code generation time is now reduced to 70 s. The simulation runs successfully. Regarding the simulation time, the model you attached has stopTime=0, which doesn't make much sense. I set StopTime = 1 and Interval = 0.01, so 100 steps are computed. Computing 100 steps takes 0.24 s on my PC, which means about 2.4 ms per step. I understand OMC can use sparse solvers while Dymola relies on tearing, on large-sized models sparse solvers could win easily.

There are a few assertions triggered at initialization, I'm not sure if this is related to comment:31, anyway the simulation starts and runs until the end.

We are working together with RTE to ensure that this kind of models is simulated as efficiently as possible automatically, without the need of setting extra flags, see, e.g., #6342. If you are interested we can get you in the loop. For the time being, please use the latest nightly build with all the latest fixes and make sure you add the custom flag annotations I mentioned above.

We are also trying to streamline the OMEdit user experience with not-so-small models. If you have some test cases that you can handle with Dymola and you would like to also handle in OpenModelica, I would suggest to collect them in a separate ticket that specifically deals with that issue.

[Francesco Casella]

Two more things. When models with over 10.000 variables are loaded, OMEdit is a bit sluggish when changing from modelling to plotting view, we should probably also try to streamline that.

Last, but not least, make sure you select Tools | Options | Simulation | Enable new frontend use in the OMC API, to get the faster possible response of the GUI. This is still not fully developed, but if it works fine it guarantees a dramatically faster response of the GUI in many cases.

[Francesco Casella]

One more thing, if you set -O0 in the "C/C++ Compiler flags (optional)" combo box in Simulation Setup | General, C-code compilation speed is dramatically improved, because the compiler avoids time-consuming optimizations.

[jean-philippe.tavella@…]

Replying to @casella, @adrpo, @adeas31:

I took in account all your remarks regarding the library PowerSysPro that can now be downloadable in version 2.1.1 from bitbucket site.

I confirm the model is loaded instantaneously. When selecting it in the package browser, it takes about 5 s on my PC before it is visualized in textual view (quite good).
check is done in 3 s (very good).
Using the flag for speeding compilation, this stage takes about 67 s (not very good).
Then the run is relativey quick.

So the main point now are:

• As RTE, EDF would prefer this kind of models can be simulated as efficiently as possible automatically, without the need of setting either annotations in the model or extra flags (e.g. for compiling).
• Compilation is to be improved when possible (I understood it will require 1 or 2 years). Dymola recently improved their compiler under EDF pressure in recent versions (Dymola 2021, 2021x) and compilation takes about 20 s now for the same model.

[Martin Sjölund]

@jean-philippe.tavella I would suggest also tagging the releases on your bitbucket so it's easy to figure out which releases are which commits (and our package manager can figure out which is the released version if it's tagged as v2.1.1).

[Francesco Casella]

Replying to jean-philippe.tavella@…:

I took in account all your remarks regarding the library PowerSysPro that can now be downloadable in version 2.1.1 from bitbucket site.

I confirm the model is loaded instantaneously. When selecting it in the package browser, it takes about 5 s on my PC before it is visualized in textual view (quite good).
check is done in 3 s (very good).
Using the flag for speeding compilation, this stage takes about 67 s (not very good).
Then the run is relativey quick.

So the main point now are:

• As RTE, EDF would prefer this kind of models can be simulated as efficiently as possible automatically, without the need of setting either annotations in the model or extra flags (e.g. for compiling).

I just reopened #4033, I think it makes sense to use -O0 as default option for C compilation.

• Compilation is to be improved when possible (I understood it will require 1 or 2 years).

There are several options of increasing difficulty here

0. Use clang instead of gcc. This was introduced in version 1.16.0 and already sped up compilation quite a bit under Windows. Also dynamic linking helped there
1. Make sure clang is as fast as possible. If I understand correctly, using -O0 halved the C-code compilation time, from two minutes to one. We'll make this default ASAP.
2. Better exploit multi-core CPUs. The C code is already spread into several different smaller .c source code files. These get compiled in parallel on multi-core CPUs, substantially speeding up this phase. Currently, there are a couple of generated C source code files that are not split appropriately, so they are not parallelized as well as they could. Doing this properly could reduce the C compilation time by a factor 2 or 3, but the backend and code generation time will remain the same, so the overall time spent before the simulation starts won't be reduced dramatically.
3. Avoid generating repeated code for instances of the same type in a model (e.g., transmission lines only differing because of numerical values of impedances). This would dramatically reduce the overall code generation and compilation time. We are working on this front, but we don't expect results for at least another one or two years. There were several presentations on this topic at the 2021 OpenModelica Workshop, please check there once the slides are published. We can discuss this further if you have specific requirements in terms of code generation and compilation performance.

Dymola recently improved their compiler under EDF pressure in recent versions (Dymola 2021, 2021x) and compilation takes about 20 s now for the same model.

I guess here you refer to the entire Modelica model compilation process, not just the C code. How long did it take in earlier versions?

[Adrian Pop]

Only 1.17+ has clang on windows, 1.16 uses the old OMDev.

[jean-philippe.tavella@…]

Replying to casella:

• Compilation is to be improved when possible (I understood it will require 1 or 2 years).

There are several options of increasing difficulty here

0. Use clang instead of gcc. This was introduced in version 1.16.0 and already sped up compilation quite a bit under Windows. Also dynamic linking helped there
1. Make sure clang is as fast as possible. If I understand correctly, using -O0 halved the C-code compilation time, from two minutes to one. We'll make this default ASAP.
2. Better exploit multi-core CPUs. The C code is already spread into several different smaller .c source code files. These get compiled in parallel on multi-core CPUs, substantially speeding up this phase. Currently, there are a couple of generated C source code files that are not split appropriately, so they are not parallelized as well as they could. Doing this properly could reduce the C compilation time by a factor 2 or 3, but the backend and code generation time will remain the same, so the overall time spent before the simulation starts won't be reduced dramatically.
3. Avoid generating repeated code for instances of the same type in a model (e.g., transmission lines only differing because of numerical values of impedances). This would dramatically reduce the overall code generation and compilation time. We are working on this front, but we don't expect results for at least another one or two years. There were several presentations on this topic at the 2021 OpenModelica Workshop, please check there once the slides are published. We can discuss this further if you have specific requirements in terms of code generation and compilation performance.

Very interesting to have options 0 & 1 activated by default in next release.
Very promising to better exploit multi-core machines, and above all to avoid generating repeated code for instances of the same type in a model. This will be very efficient for electrical models where we have very few different classes and lots of instances of these classes.

Dymola recently improved their compiler under EDF pressure in recent versions (Dymola 2021, 2021x) and compilation takes about 20 s now for the same model.

I guess here you refer to the entire Modelica model compilation process, not just the C code. How long did it take in earlier versions?

I cannot answer this question as we previsouly used a commercial library for electrical models (library EPSL from Modelon, which is now a Dassault Systèmes library). Nevertheless I attach an extract of a paper giving an overview of their performanace improvements (note these improvements were partly based on rewriting some components in EPSL library too).

[Adrian Pop]

You can do 0 and 1 even now to test:

• for 0 you would need the nightly builds (or 1.17 when I manage to build it).
• for 1 press S in the model view and add -O0 in "C/C++ compiler flags (Optional)".

[jean-philippe.tavella@…]

Replying to adrpo:

You can do 0 and 1 even now to test:

• for 0 you would need the nightly builds (or 1.17 when I manage to build it).
• for 1 press S in the model view and add -O0 in "C/C++ compiler flags (Optional)".

Already sucessufully done.
What is the planning date for 3 (and 2 too)?

[Francesco Casella]

2. could be easily done this year, if that is interesting for EDF. We would need some challenging test cases, so we can make sure we are splitting the right files, otherwised it would be a waste of time.

3. as I wrote in comment:37 is more medium-long term development, we have several activities running that may start to deliver something by the end of 2021.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

I cannot answer this question as we previsouly used a commercial library for electrical models (library EPSL from Modelon, which is now a Dassault Systèmes library). Nevertheless I attach an extract of a paper giving an overview of their performanace improvements (note these improvements were partly based on rewriting some components in EPSL library too).

I see huge intialization times, sometimes exceeding 1 hour. This may be due to the use of dense solvers. Have you tried with OpenModelica, using the flags I mentioned in comment:32?

[jean-philippe.tavella@…]

Replying to casella:

Replying to jean-philippe.tavella@…:

I cannot answer this question as we previsouly used a commercial library for electrical models (library EPSL from Modelon, which is now a Dassault Systèmes library). Nevertheless I attach an extract of a paper giving an overview of their performanace improvements (note these improvements were partly based on rewriting some components in EPSL library too).

I see huge intialization times, sometimes exceeding 1 hour. This may be due to the use of dense solvers. Have you tried with OpenModelica, using the flags I mentioned in comment:32?

Not possible as EPSL cannot be used in OM due to the fact it is a DS commercial library.

[jean-philippe.tavella@…]

Replying to casella:

2. could be easily done this year, if that is interesting for EDF. We would need some challenging test cases, so we can make sure we are splitting the right files, otherwised it would be a waste of time.

3. as I wrote in comment:37 is more medium-long term development, we have several activities running that may start to deliver something by the end of 2021.

EDF can supply test cases, for example in the context of european project EMBrACE to scale OM with large models (multi-mode or not).
No problem for electrical models.
For thermal building models, I indicate that a study on the BuildSysPro / OpenModelica compatibility will be presented at the next OpenModelica conference on Feb 2nd, by some colleagues of mine. Some issues have already been reported and are to be solved before or in parallel.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

I see huge intialization times, sometimes exceeding 1 hour. This may be due to the use of dense solvers. Have you tried with OpenModelica, using the flags I mentioned in comment:32?

Not possible as EPSL cannot be used in OM due to the fact it is a DS commercial library.

Of course, sorry. On the other hand, I understand DS is introducing sparse solvers in Dymola, maybe you should ask about them.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

EDF can supply test cases, for example in the context of european project EMBrACE to scale OM with large models (multi-mode or not).
No problem for electrical models.

That would be great. Even better if those models are available on a public GIT repository, that we can link to our testsuite. Are they?

For thermal building models, I indicate that a study on the BuildSysPro / OpenModelica compatibility will be presented at the next OpenModelica conference on Feb 2nd, by some colleagues of mine. Some issues have already been reported and are to be solved before or in parallel.

Saw that, thanks. I think we should try as much as possible to put concrete, real-life, challenging example cases in our testsuite, so we can play around with them and try to improve the performance. Having them in the testsuite allows a more structured activity. Would that be possible?

[jean-philippe.tavella@…]

Replying to casella:

Saw that, thanks. I think we should try as much as possible to put concrete, real-life, challenging example cases in our testsuite, so we can play around with them and try to improve the performance. Having them in the testsuite allows a more structured activity. Would that be possible?

I think so.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

Replying to casella:

Saw that, thanks. I think we should try as much as possible to put concrete, real-life, challenging example cases in our testsuite, so we can play around with them and try to improve the performance. Having them in the testsuite allows a more structured activity. Would that be possible?

I think so.

OK. Then the requirements are the following:

• The library with the examples should be stored on a git repository (preferably a public one)
• All the models to be simulated should have an experiment(StopTime) annotation
• You just send us the URL of the repository and we add it to the testsuite

[anonymous]

Replying to casella:

Of course, sorry. On the other hand, I understand DS is introducing sparse solvers in Dymola, maybe you should ask about them.

I think Dymola implements a sparse linear solver, may be (not sure) superLU.

[jean-philippe.tavella@…]

Replying to casella:

OK. Then the requirements are the following:

• The library with the examples should be stored on a git repository (preferably a public one)
• All the models to be simulated should have an experiment(StopTime) annotation
• You just send us the URL of the repository and we add it to the testsuite

I think it as possible but not before the end of current French project ModeliScale (>July 2021) where we are working on a medium size energetic system involving electrical grid, buildings, DHN and CHP with our *SysPro libraries.
Audrey Jardin and Daniel Bouskela have also to agree on that.

[Francesco Casella]

Replying to jean-philippe.tavella@…:

I think it as possible but not before the end of current French project ModeliScale (>July 2021) where we are working on a medium size energetic system involving electrical grid, buildings, DHN and CHP with our *SysPro libraries.

Sounds good, we are not in a hurry. Very nice to have publicly available test cases of such combined thermal-electrical systems. BTW, we have some ideas about the use of multi-rate solvers, that could be very useful to substantially improve the simulation performance in such cases of systems with widely different time scales, those could be again very good test cases.

Audrey Jardin and Daniel Bouskela have also to agree on that.

OK. I guess they will be fine with it.

[Francesco Casella]

@jean-philippe, I think we can now close this ticket before it overcomes some Dumas' novel in length :)

• Regarding loading, I think we have decent performance now.
• Regarding automatic selection of the most efficient compilation and simulation options to get fast simulation, see #6340 and #6342, I put you in cc: there. Until this is not fixed, make sure you select --daeMode during compilation and -nls=kinsol during simulation for the best performance of models including AC power systems.
• Regarding faster compilation, see #6367.
• Regarding testing of ModeliScale models, please come back to us when they can be made available for testing.

Thanks for the feedback!

[jean-philippe.tavella@…]

Replying to casella:

Agreed :-)

[Adrian Pop]

@jean-philippe, you changed the PowerSysPro lib in your latest commit and you broke the OpenModelica loading:
​https://bitbucket.org/simulage/powersyspro/commits/234760f297b7d736e28d4978692e1bd677a5a22d
You renamed PowerSysPro.mo to PowerSysPro-2.1.1.mo but this is not conform to the spec, see here:
​https://specification.modelica.org/maint/3.4/Ch18.html#mapping-of-versions-to-file-system
Just rename it again to: PowerSysPro 2.1.1.mo and it should be fine.

[jean-philippe.tavella@…]

Replying to adrpo:

Just rename it again to: PowerSysPro 2.1.1.mo and it should be fine.

Sorry done.

[Francesco Casella]

Added a paragraph on this to the Porting section of the User's Guide, see ​PR 7140.