IDA solver in DAE mode fails in models with events

[Francesco Casella]

When simulating the models ManyEvents and ManyEventsManyConditions of the ScalableTestSuite, using the --daeMode compile flag and the -daeMode runtime flag, the following error is obtained

stdout            | warning | -daeMode flag is used, the model is not compiled in DAE mode. See compiler flag: +daeMode. Continue as usual.
*** Error in `./ScalableTestSuite.Elementary.WhenEvents.Verification.ManyEventsManyConditions':
malloc(): memory corruption: 0x000000000244fab0 ***

See ​https://test.openmodelica.org/libraries/ScalableTestSuite_Experimental/BuildModelRecursive.html

[Adrian Pop]

Also it seems that we get the message:

stdout            | warning | -daeMode flag is used, the model is not compiled in DAE mode. See compiler flag: +daeMode. Continue as usual.

for *all* the models even if they are compiled with --daeMode.

I had to manually kill some of the models in the LibraryExperimental because they hanged.
I will come back with a list with them as I started the LibraryExperimental again.
I also moved the LibraryExperimental Hudson job on the new server (libraries.openmodelica.org).

[Adrian Pop]

Model: ScalableTestSuite.Elementary.WhenEvents.ScaledExperiments.ManyEventsManyConditions_N_8000 hangs here:

#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f57ef85f64a in __GI___libc_malloc (bytes=140015660825376, bytes@entry=56) at malloc.c:2912
#2  0x00007f57f1202567 in _dl_map_object_deps (map=map@entry=0x7f57f13fdad0, preloads=preloads@entry=0x0, npreloads=npreloads@entry=0, trace_mode=trace_mode@entry=0,
    open_mode=open_mode@entry=-2147483648) at dl-deps.c:499
#3  0x00007f57f1209412 in dl_open_worker (a=a@entry=0x367dbc0) at dl-open.c:280
#4  0x00007f57f1204394 in _dl_catch_error (objname=objname@entry=0x367dbb0, errstring=errstring@entry=0x367dbb8, mallocedp=mallocedp@entry=0x367dbaf,
    operate=operate@entry=0x7f57f1209300 <dl_open_worker>, args=args@entry=0x367dbc0) at dl-error.c:187
#5  0x00007f57f1208bd9 in _dl_open (file=0x7f57ef967ec6 "libgcc_s.so.1", mode=-2147483647, caller_dlopen=0x7f57ef8f11c5 <init+21>, nsid=-2, argc=<optimized out>,
    argv=<optimized out>, env=0x7fffec53b478) at dl-open.c:660
#6  0x00007f57ef91eddd in do_dlopen (ptr=ptr@entry=0x367dde0) at dl-libc.c:87
#7  0x00007f57f1204394 in _dl_catch_error (objname=0x367ddd0, errstring=0x367ddd8, mallocedp=0x367ddcf, operate=0x7f57ef91eda0 <do_dlopen>, args=0x367dde0) at dl-error.c:187
#8  0x00007f57ef91ee94 in dlerror_run (args=0x367dde0, operate=0x7f57ef91eda0 <do_dlopen>) at dl-libc.c:46
#9  __GI___libc_dlopen_mode (name=name@entry=0x7f57ef967ec6 "libgcc_s.so.1", mode=mode@entry=-2147483647) at dl-libc.c:163
#10 0x00007f57ef8f11c5 in init () at ../sysdeps/x86_64/backtrace.c:52
#11 0x00007f57efbb3ad9 in __pthread_once_slow (once_control=0x7f57efba3170 <once>, init_routine=0x7f57ef8f11b0 <init>) at pthread_once.c:116
#12 0x00007f57ef8f12e4 in __GI___backtrace (array=<optimized out>, size=<optimized out>) at ../sysdeps/x86_64/backtrace.c:105
#13 0x00007f57f0ebb732 in handler () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#14 <signal handler called>
#15 _int_malloc (av=av@entry=0x7f57efb9fb20 <main_arena>, bytes=bytes@entry=40) at malloc.c:3728
#16 0x00007f57ef85f5a4 in __GI___libc_malloc (bytes=40) at malloc.c:2914
#17 0x00007f57eee1fa20 in NewSparseMat () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#18 0x00007f57eee2117e in SlsAddMat () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#19 0x00007f57f0f74e16 in jacobianSparseNum () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#20 0x00007f57eee178e0 in IDAKLUSetup () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#21 0x00007f57eedfd797 in IDASolve () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#22 0x00007f57f0f76378 in ida_solver_step () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#23 0x00007f57f0f624e1 in solver_main_step () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#24 0x0000000000ea54ee in ScalableTestSuite_Elementary_WhenEvents_ScaledExperiments_ManyEventsManyConditions_N_8000_performSimulation ()
#25 0x00007f57f0f63dd7 in solver_main () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#26 0x00007f57f0f95691 in callSolver(DATA*, threadData_s*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, double, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, char const*) ()
   from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#27 0x00007f57f0f94bad in startNonInteractiveSimulation ()
   from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#28 0x00007f57f0f96f63 in _main_SimulationRuntime () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#29 0x00000000011b0606 in main ()

[Adrian Pop]

Model: ScalableTestSuite.Elementary.WhenEvents.Verification.ManyEvents
hangs here:

#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f07feb6d64a in __GI___libc_malloc (bytes=139672318303008, bytes@entry=56) at malloc.c:2912
#2  0x00007f0800510567 in _dl_map_object_deps (map=map@entry=0x7f080070bad0, preloads=preloads@entry=0x0, npreloads=npreloads@entry=0, trace_mode=trace_mode@entry=0,
    open_mode=open_mode@entry=-2147483648) at dl-deps.c:499
#3  0x00007f0800517412 in dl_open_worker (a=a@entry=0x1599bc0) at dl-open.c:280
#4  0x00007f0800512394 in _dl_catch_error (objname=objname@entry=0x1599bb0, errstring=errstring@entry=0x1599bb8, mallocedp=mallocedp@entry=0x1599baf,
    operate=operate@entry=0x7f0800517300 <dl_open_worker>, args=args@entry=0x1599bc0) at dl-error.c:187
#5  0x00007f0800516bd9 in _dl_open (file=0x7f07fec75ec6 "libgcc_s.so.1", mode=-2147483647, caller_dlopen=0x7f07febff1c5 <init+21>, nsid=-2, argc=<optimized out>,
    argv=<optimized out>, env=0x7ffd6f0654b8) at dl-open.c:660
#6  0x00007f07fec2cddd in do_dlopen (ptr=ptr@entry=0x1599de0) at dl-libc.c:87
#7  0x00007f0800512394 in _dl_catch_error (objname=0x1599dd0, errstring=0x1599dd8, mallocedp=0x1599dcf, operate=0x7f07fec2cda0 <do_dlopen>, args=0x1599de0) at dl-error.c:187
#8  0x00007f07fec2ce94 in dlerror_run (args=0x1599de0, operate=0x7f07fec2cda0 <do_dlopen>) at dl-libc.c:46
#9  __GI___libc_dlopen_mode (name=name@entry=0x7f07fec75ec6 "libgcc_s.so.1", mode=mode@entry=-2147483647) at dl-libc.c:163
#10 0x00007f07febff1c5 in init () at ../sysdeps/x86_64/backtrace.c:52
#11 0x00007f07feec1ad9 in __pthread_once_slow (once_control=0x7f07feeb1170 <once>, init_routine=0x7f07febff1b0 <init>) at pthread_once.c:116
#12 0x00007f07febff2e4 in __GI___backtrace (array=<optimized out>, size=<optimized out>) at ../sysdeps/x86_64/backtrace.c:105
#13 0x00007f08001c9732 in handler () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#14 <signal handler called>
#15 _int_malloc (av=av@entry=0x7f07feeadb20 <main_arena>, bytes=bytes@entry=40) at malloc.c:3417
#16 0x00007f07feb6d5a4 in __GI___libc_malloc (bytes=40) at malloc.c:2914
#17 0x00007f07fe12da20 in NewSparseMat () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#18 0x00007f07fe12f17e in SlsAddMat () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#19 0x00007f0800282e16 in jacobianSparseNum () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#20 0x00007f07fe1258e0 in IDAKLUSetup () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#21 0x00007f07fe10b797 in IDASolve () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libsundials_idas.so.0
#22 0x00007f0800284378 in ida_solver_step () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#23 0x00007f08002704e1 in solver_main_step () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#24 0x0000000000419cfe in ScalableTestSuite_Elementary_WhenEvents_Verification_ManyEvents_performSimulation ()
#25 0x00007f0800271dd7 in solver_main () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#26 0x00007f08002a3691 in callSolver(DATA*, threadData_s*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, double, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, char const*) ()
   from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#27 0x00007f08002a2bad in startNonInteractiveSimulation ()
   from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#28 0x00007f08002a4f63 in _main_SimulationRuntime () from /var/lib/hudson/slave/workspace/LibraryExperimental/OpenModelica/build/lib/x86_64-linux-gnu/omc/libSimulationRuntimeC.so
#29 0x0000000000422ffc in main ()

[Adrian Pop]

The new server seems to have issues (gets stuck) so I moved the Hudson job back on master.

[Willi Braun]

This works for me, also in the last run on the server the models are working:
(see ​ScalableTestSuite_Experimental)
So probably it's fixed in the meanwhile.