Initialization solver hangs or fails at start of homotopy process without a clue what it is doing

[casella]

Please try simulating PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization. If you add lv=LOG_NLS_V,LOG_INIT_V and simulate it, the solver outputs

...
Global homotopy with equidistant step size started

The homotopy path will be exported to PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization_equidistant_global_homotopy.csv.

then it hangs indefinitely at 0% progress. What I find weird is that I get no log output at all. Any idea what is it doing?

[casella]

Probably related to #5795, as it happens exactly in those cases having if initial() clauses within initial equations

[casella]

The root cause of the issue is reported in #5805, please refer to that. I'll keep this ticket open to see if our test case works once #5805 and #5795 are resolved.

[casella]

Retargeted to 1.17.0 after 1.16.0 release

[casella]

@AnHeuermann, despite all other related issues having been fixed, this is still there with the latest 1.18.0 nightly.

[casella]

Probably related to #6348.

[casella]

Replying to casella:

Probably related to #6348.

​PR 7207 fixed #6348 but did not fix this issue. This is less urgent, but it is worth investigating it with gdb as well.

[adrpo]

I tried to reproduce this from command line and have no luck, it just generates a 38MB log file.
Do I need to do it in OMEdit? Where should I add LOG_NLS_V,LOG_INIT_V, in the __OpenModelica_simulationFlags?

[casella]

Replying to adrpo:

I tried to reproduce this from command line and have no luck, it just generates a 38MB log file.

Too bad, apparently the command line does not set the extra bits :)

Do I need to do it in OMEdit?

Yes

Where should I add LOG_NLS_V,LOG_INIT_V, in the __OpenModelica_simulationFlags?

Simulation Setup | Simulation flags, tick the corresponding checkboxes

[casella]

Retargeted to 1.18.0 after 1.17.0-dev.beta2 release

[adrpo]

The executable consumes *A LOT* of memory, until I took my time to attach with GDB it reached 9GB. This is the backtrack:

(gdb) bt
#0  0x00007ffc1fce2192 in ?? ()
   from C:\home\adrpo33\dev\OpenModelica\build\bin\libstdc++-6.dll
#1  0x00000000004de965 in messageXMLTCP(int, int, int, char*, int, int const*)
    ()
#2  0x000000000049e10d in warningStreamPrint ()
#3  0x00000000004d392d in kinsolInfoHandlerFunction ()
#4  0x0000000063184484 in KINPrintInfo (kin_mem=kin_mem@entry=0x4ef939d0,
    info_code=<optimized out>, info_code@entry=9,
    module=module@entry=0x631b1220 "KINSOL",
    fname=0x631b196e "KINLinesearch",
    msgfmt=0x631b197c "fnorm = %15.8le   f1norm = %15.8le   alpha_cond = %15.8le  lam = %15.8le")
    at C:/home/adrpo33/dev/OpenModelica/OMCompiler/3rdParty/sundials-5.4.0/src/kinsol/kinsol.c:2167
#5  0x0000000063183669 in KINLineSearch (kin_mem=0x4ef939d0,
    fnormp=<optimized out>, f1normp=<optimized out>,
    maxStepTaken=<optimized out>)
    at C:/home/adrpo33/dev/OpenModelica/OMCompiler/3rdParty/sundials-5.4.0/src/kinsol/kinsol.c:1630
#6  KINSol (kinmem=0x4ef939d0, u=<optimized out>,
    strategy_in=<optimized out>, u_scale=<optimized out>, f_scale=0x4ef6a710)
    at C:/home/adrpo33/dev/OpenModelica/OMCompiler/3rdParty/sundials-5.4.0/src/kinsol/kinsol.c:565
#7  0x00000000004c6a89 in nlsKinsolSolve ()
#8  0x00000000004a68e4 in solveNLS ()
#9  0x00000000004a6f5c in solve_nonlinear_system ()
#10 0x000000000043fbf0 in TwoGeneratorsLocalInitialization_eqFunction_1060 ()
#11 0x00000000004440fb in TwoGeneratorsLocalInitialization_functionInitialEquations_lambda0 ()
#12 0x00000000004d782a in symbolic_initialization ()
#13 0x00000000004d6fc0 in initialization ()
#14 0x00000000004b5356 in initializeModel ()
#15 0x00000000004b6648 in solver_main ()
#16 0x00000000004dbafb in callSolver(DATA*, threadData_s*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, char const*) ()
#17 0x00000000004dabca in startNonInteractiveSimulation ()
#18 0x00000000004de31f in _main_SimulationRuntime ()
#19 0x000000000040eb98 in main ()

The exe is not really hanging, it just writes a lot of info to the warning stream and I'm not really sure how that is implemented. Is the warning stream buffered in memory until a step is done, if so that might be the issue? I thought is sent via tcp as xml to OMEdit but somehow that doesn't seem to happen in this case, we should see OMEdit consume the stream.

[adrpo]

Ok, this example generates *a lot* of info, I ran it like this:

adrpo33@ida-0030 MINGW64 /c/temp/PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization
# ./TwoGeneratorsLocalInitialization.exe -override=startTime=0,stopTime=1,stepSize=0.002,tolerance=1e-6,solver=dassl,outputFormat=mat,variableFilter=.* -r=C:/temp/PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization/TwoGeneratorsLocalInitialization_res.mat -nls=kinsol -w -lv=LOG_INIT_HOMOTOPY,LOG_INIT_V,LOG_NLS_V,LOG_STATS -homotopyOnFirstTry -inputPath=C:/temp/PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization -outputPath=C:/temp/PowerGrids.Examples.Tutorial.IslandOperation.TwoGeneratorsLocalInitialization > trace.txt 2>&1

and stopped it when trace.txt reached 1GB, it just fills my ssd with logs.

The only thing that OMEdit has in plus is -port=54016 -logFormat=xmltcp. Somehow I don't think that sending GBs of logs via xmltcp would work nicely.

[adrpo]

Is OMEdit suppose to handle GB of log info via xmltcp?

[casella]

Not really, the idea is that whenever the log file exceeds 10 MB, OMEdit stops receiving it and the log is only saved to a file, to which a link in the OMEdit simulation output window is provided.

I thought this was solved already in #4475. Apparently, that solution was not thourough enough.

[casella]

Ticket retargeted after milestone closed

[casella]

1.18.0 blocker tickets moved to 1.19.0