Too many heap sections

[ceraolo]

In recent weeks I've used OMC through OMEdit a lot.
Unfortunately, very frequently I had a program crash that was immediately preceded by just a message saying "too many heap sections".
I remember that in the recent past an error notification was sent to program developers upon program crash.
I seems like in more recent builds this automatic notification has been dropped. I currently use r25870.

Is it possible to activate this notification again so that more information is available when that "too many heap sections" crash occurs?
Or there is already some idea on what could cause this issue?

[adrpo]

That message is from the garbage collector library that we use.
I'll investigate if we can fix it somehow.
And the crash reports did not work for a while but are working now as far as I know.
I guess it also depends how the program crashes. In some cases the crash report might
not be possible to run and collect the information.

[ceraolo]

Yes, the error is related to the garbage collector.
In fact, It has occurred again and I saw in the message box header the text: "Fatal error in GC"

I must add that I've used, for teaching, r25516 (win version) MANY times and never had this issue. Therefore I'm pretty sure that it has been introduced later.

As I said I've observed the crash several times with r25870 (for win) and later releases.

[sjoelund.se]

See ​https://github.com/OpenModelica/OMCompiler-3rdParty/blob/master/gc/doc/README.environment for how to control garbage collection.

Note that at least on Linux we disabled things like --enable-munmap=5 that is enabled on Windows. On Windows, we very similar problem (reaching maximum number of munmap for a process, etc).

Am I correct in assuming the error occurs on Windows?

As for OMEdit crash reports, it only catches some errors. If libgc calls abort, it would just fail here. And now OMEdit and omc are the same process, so if omc aborts, OMEdit dies too.

[ceraolo]

Yes, I had the error on Windows.

[ceraolo]

I can be more precise now.
According to my tests, the nasty "Too many heap sections" occurs rather often on Windows, starting from r25613.
I confirm that it never occurred to me in r25516 (windows version as well).

So, in case there is interest to find the problem, it is now traced down to a limited range of versions.

[adrpo]

You mean that it happens in r25613 and it doesn't happen in r25516?
That's a lot of revisions. I had a look at the range and there are modifications to OMEdit and the compiler library (new tarjan, etc) so I cannot say directly which revision is to blame.

Do you have any model or sequence of commands I could use to reproduce it?
When this happens you could zip directory %TEMP%\OpenModelica\OMEdit (I'm mostly interested in omeditcommunication.log and omeditcommands.mos) and attach it here.

[adrpo]

I could reproduce the bug with the latest nightly-build by running checkModel on Spice3 FourBitAdder several times. I'll see if I can do something about it.

[adrpo]

I played a bit with how GC gives back memory to the OS in ​https://github.com/OpenModelica/OMCompiler/commit/0fa54ddeaf2fd60dc96ebcfd0abae2410f1a23cc
As far as I can see the problem does not appear so often.
You could try the next nightly-build and let me know if it works for you.

[ceraolo]

I will download during the next days.
Typically I spend a lot of time with OM during week-ends, and I will extensively use the new OM next week-end at the latest.
If it survives 2-3 hours of continuous work, for me it will be ok.

Just for completeness, I have r25613 on my Macbook air. It works wonderful!
It is very fast and very reliable. Never seen the "Too many heap sections" nasty window, never experienced the sluggishness that occurs windows a few minutes before the "Too many heap sections" message.

So, any solution that is used for Mac to manage GC, and is transferable onto Win, I suppose, is good candidate for Win systems as well.
I mention this just because from you message I envisage that you are not totally sure of the effectiveness of your recent fix.

[adrpo]

I guess your Mac one is 64 bit. That helps a lot as pointers are 64 bit wide and not 32 bit.

[ceraolo]

I had time to use OM bf6ce68 some hours in the last couple of days.
The result is that the "Too many heap sections" issue has occurred to me an average of once per hour of activity (changing models, switching perspectives, running, loading and unloading models and results, etc.).
This is not good for a stable release, but is acceptable for a nightly build.
As an OM supporter I expect that the next release will have this issue solved more radically than now is.

I understand that this issue is not totally related with the 32 bit address space of 32bit Windows (since, for Windows, releases up to r25516 did not have it), but that I can infer from you messages that switching to 64 bit should solve it totally.

[ceraolo]

I've made some tests with today's nightly build for Windows. The executable is named
OpenModelica-v1.9.3-dev-413-g8762871.exe, even though OMEdit reports
OpenModelica v1.9.3-dev-956-g91eaa01

Twice after around a quarter of an hour of activity I got the "too many heap sections" message and a program crash.
I understand that we must wait for the 64-bit Win version for a solution of this issue.
Is it possible to have a guess on when this version is going to be issued? Or, if there is some other way under way to overcome this ticket's issue?

Thanks

[adrpo]

ceraolo, you could give this one (a 64bit OMEdit) a try:
​https://build.openmodelica.org/omc/mingw64/OMEdit64bit.zip
and let me know how it goes. Hopefully this issues should
not appear that often.

[ceraolo]

I understand from your message that this zip can be integrated into an existing OM installation, even though it sounds strange since the nightly builds should be 32 bit, AFAIK.

Can you please give me short advice on how to integrate this zip into an existing OM installation?
At present I have, I think, v1.9.3-dev-413-g8762871 (but OMEdit reports v1.9.3-dev956-g91eaa01).

[adrpo]

You cannot integrate it into an existing installation.
Just expand the zip directory someplace and run OMEdit from it.
It will only use your existing OpenModelica installation to build
the generated C code.

I hope I can integrate this into the normal installer later this
week but I wanted to give it away so people can do some preliminary
testing.

[ceraolo]

I was unable to do what you suggest, sorry. I get a message saying:

This application failed to start because it could not find or load the Qt platform plugin "windows"

I suspect it is best for me to wait until when the normal installer with this enhancement is ready.

[sjoelund.se]

Moved to new milestone 1.9.4

[ceraolo]

I've tried the new rel 1.9.3.
The "Too many heap sections" appears to be much more frequent than it used to.
Cannot be sure, because I don't remember exactly what I was doing when I tried some months ago.
Today OMEdit crashed with that message every 10 minutes as an average.

[adrpo]

Unfortunately we could not make the 64 bit version for Windows available in time for 1.9.3 because of the big number of changes. However, now that we released 1.9.3 I can commit my changes and we can switch to 64 bit versions of OpenModelica which should fix this problem.
I plan to have a 64 bit nightly-build available by the end of next week.

[ceraolo]

Wow!
This is really good news!

[ceraolo]

Somewhere I read that the C++ runtime produced by OM does not use garbage collector; it exploits object destructors instead.
If I understand well, so using C++ runtime would help on 32-bit win systems to avoid this "too many heap sections" issue.
Or I am missing something?

[sjoelund.se]

The FMUs also do not use garbage collector (since a recent nightly build).

[adrpo]

The C++ runtime is only for simulation.
OMC is using MetaModelica as implementation language and the MetaModelica
runtime to run which uses the Boehm GC and there is no way around it.
Unfortunately I've been busy with other stuff and the 64 bit Windows has progressed rather slow but hopefully it will be available soon.

[anonymous]

I solved the problem in windows by adding an environment variable.
Variable: GC_Free_SPACE_DIVISOR, Value: 32

[ceraolo]

Today I tried this environment variable GC_Free_SPACE_DIVISOR.
At the end of one hour's work I still got the "too many heap sections" GC message and the system crash.
I cannot be sure whether setting the environment variable has enlarged the time after which the message (and crash) appears, because that time is not always the same, even with a given system configuration, obviously, and I could not succeeded in creating a repetitive sequence to make it occur.
I used OpenModelica v1.9.4-dev-490-g44a6be4

[ceraolo]

This issue is well and alive also in rel OM v1.9.4-dev.beta1-42-g0c39bf6.

Just to let you know, since it would be nasty it v1.9.4 is released, and under windows the user receives very frequent crashes.

[adrpo]

I made a 64 bit OM installer. This is the first version and is rather big 1.7GB and expands to about 7GB.
​https://build.openmodelica.org/omc/builds/windows/nightly-builds/64bit/
casella and ceraolo, could you give it a try?

[adrpo]

Replying to adrpo:

I made a 64 bit OM installer. This is the first version and is rather big 1.7GB and expands to about 7GB.
​https://build.openmodelica.org/omc/builds/windows/nightly-builds/64bit/
casella and ceraolo, could you give it a try?

I just tested a bit and the installer is not ready yet. I'll try to make another one tomorrow.

[ceraolo]

I made some preliminary tests on the 64 bit version.
A few comments:
1) I could not run tests.
There was some path issue. When running a model an error message complains about paths. Note that it mentions the OMEdit Working directory, of which I use a personalised version. However, even after blanking that custom path (in "in OMEdit|Options|General|Working directory"), the same error message blocks execution (still quoting the path I've deleted)
2) appearance
The appearance of the diagram view of OMEdit is changed. All gridlines have become thicker, too thick, indeed.
3) timing
According to my very first test, opening a model containing several submodels in a package requires roughly twice the time (25s instead of 12). I can supply the mo file, just in case.

[adrpo]

Replying to ceraolo:

I made some preliminary tests on the 64 bit version.
A few comments:
1) I could not run tests.
There was some path issue. When running a model an error message complains about paths. Note that it mentions the OMEdit Working directory, of which I use a personalised version. However, even after blanking that custom path (in "in OMEdit|Options|General|Working directory"), the same error message blocks execution (still quoting the path I've deleted)
2) appearance
The appearance of the diagram view of OMEdit is changed. All gridlines have become thicker, too thick, indeed.
3) timing
According to my very first test, opening a model containing several submodels in a package requires roughly twice the time (25s instead of 12). I can supply the mo file, just in case.

Thanks ceraolo for the testing! Do you by any chance have OMDEV environment variable defined?
If so, please remove it when you test with this version as it will cause issues with the compilation. Good to know about the other issues like gridlines and performance. This is build using a newer GCC (5.3 instead of 4.4) and maybe we haven't yet optimized it for it.

[sjoelund.se]

Replying to adrpo:

Thanks ceraolo for the testing! Do you by any chance have OMDEV environment variable defined?
If so, please remove it when you test with this version as it will cause issues with the compilation. Good to know about the other issues like gridlines and performance. This is build using a newer GCC (5.3 instead of 4.4) and maybe we haven't yet optimized it for it.

Or it is that the word size is twice as large and most of the time opening a model is parsing, which means memory allocation. With ~2x memory allocation comes ~2x performance loss.

Byte sizes 32->64-bit:
Integer: 4->8
Boxed Integer: 4->8
Double: 8
Boxed double: 8->12
Strings: 4+4*(len/4)->8+8*(len/8)
Tuples: 4*(1+n)->8*(1+n)
Uniontypes: 4*(2+n)->8*(2+n)
Lists: 12->24
Option: 8->16

[sjoelund.se]

You should be able to tell by using -d=execstat to measure memory allocated by loadModel+getErrorString

[adrpo]

It might also be that the performance issues come from the special flags we use to compile with GCC 5.x (-fno-ipa-pure-const).

[casella]

It would be good to compare the performance of 64-bit OMEdit on Windows to the performance of 64-bit OMEdit under linux. Andrea, can you try that?

[ceraolo]

I must add that I made the tests using two sligtly different computers. Actual results are:

• slower PC OM32: 12.0 s
• faster PC OM64 18.9 s.

Then I made some compensation to try to take into account the computer speed difference.
I know that this is unreliable.
Is there a way to keep on the same computer both OM32 and OM64? It would ease comparisons a lot.

[adrpo]

Replying to ceraolo:

I must add that I made the tests using two sligtly different computers. Actual results are:

• slower PC OM32: 12.0 s
• faster PC OM64 18.9 s.

Then I made some compensation to try to take into account the computer speed difference.
I know that this is unreliable.
Is there a way to keep on the same computer both OM32 and OM64? It would ease comparisons a lot.

Currently is not possible to install both 32 and 64 bit on the same computer. I'll see what I can do about that.

[casella]

Adrian, don't waste time on this. Virtual machines are the way to handle these comparisons. Incidentally, this would also allow to compare Windows 32, Windows 64 and Linux 64 on the same hardware.

[ceraolo]

Replying to adrpo:

[...] Do you by any chance have OMDEV environment variable defined?

No I don't. The OM-related environment variables I have are:
OPENMODELICAHOME C:\OpenModelica1.9.4.-dev.beta2\
OPENMODELICALIBRARY c:\Openmodelica1.9.4.-dev.beta2\lib\omlibrary

I must add that I also have the following one (as suggested by a user to reduce the issue "too many heap sections":
GC_Free_SPACE_DIVISOR 32

Replying to casella
Yes, I can install the 32 bit version in an Oracle's VB. Do you think in-VB times and in-Win times can be reliably compared to each other?
I've always suspected this mustbe true, except for the programs that make substantial use of I/O (games, for instance that make heavy use of graphical cards and are substantially slower from inside a VM), but I never had precise information on this.

Finally: How can you make a 64 bit OS to run in a VM? AFAIK Oracle's Virtual Box only allows 32 bit OS's. Maybe you think of different virtualisation SW.

[Christoph <buchner@…>]

I don't know about Windows guests, but I have Virtualbox with an Ubuntu 64bit Guest on a 64bit Windows host, so your information may be outdated.

[sjoelund.se]

It's very simple to install 2 copies of OM: First install one, rename the directory. The install the other.
Unset all OM env.vars and let omc.exe figure it out by itself (if there is no OPENMODELICAHOME, it just works...)

[ceraolo]

Installing OM32 and OM64 on the same PC, without any VM following Martin's instructions worked!

No I can update my timing info, for opening the file FullVehicles.PSEcu2 of the enclosed EHPowerTrain.mo
1) OMEdit 32 bit Connected to OpenModelica v1.9.4-dev.beta2-140-g6f118e9: 19.19 s
2) OMEdit 64 bit Connected to OpenModelica v1.9.4-dev.beta2-172-ge58d777: 22.29 s
So there are not significant differences between 32 and 64 bits.

The difference that I noticed in my previous message, then, I think is more related to the fact that the faster timing was obtained with an older OM (that resulted faster for this test).

BTW, I miss indication "32bit" or "64bit" in the indication of versions from inside OMEdit Help|about dialog. Is it possible to add this indication?

[sjoelund.se]

Milestone pushed to 1.9.5

[sjoelund.se]

Milestone renamed

[adrpo]

Fixed via #2578.

[adrpo]

See also #3868 32bit OMEdit now works much better.

[sjoelund.se]

Milestone deleted