SimCode generation time does not scale properly with the system size

[Francesco Casella]

Consider the results on the compilation of the models ScalableTestSuite.Electrical.TransmissionLine.ScaledExperiments.TransmissionLineModelica_N_XX reported here:

​https://test.openmodelica.org/libraries/ScalableTestSuite_Experimental/BuildModelRecursive.html

It is immediate to notice that the SimCode time scales approximately with the square of the sytem size. This is not right, as we might expect that the amount of code that describes these models is roughly proportional to their size. For large models, this can lead to prohibitevely high times.

It might be the case that the implementation of the functionalities in SimCode contains some double recursion that originates this behaviour.

Please investigate if this is the case and, if so, try to reimplement the functions avoiding the double recursion and the quadratic growth of the execution time.

[Francesco Casella]

I see there's some improvement after ​PR447. However, I still see almost quadratic scaling, see e.g. the ScalableTestSuite.Mechanical.Strings.ScaledExperiments.StringModelica_N_XX models.

[Willi Braun]

Yeah, that's also the reason I haven't closed it ;).
However, I guess I found the other place already also fixed is almost.
So I'll hope to push quite soon and then close than.

[Willi Braun]

An further improvement should result from ​PR473. Let's see and wait for the results from the coverage run.

[Henning Kiel]

Using callgrind (OSX) I see that for simple models already at least 1/3 of the time is spent with GC. And quickly several GB of RAM were allocated and freed by GC.

One possible candidate is the massive use of listAppend, which for each call duplicates the first list and appends the second. If that is used recursively/in a loop the memory consumption grows quadratically (and GC time should also grow like this).

To reduce the overhead, I implemented a function listAppendTail() which assumes that the first list is sort of "input output", so that it expects more calls appending to that list.

function test_appendListTail<T>
  input list<list<T>> inLst;
  output list<T> outLst = {};
protected
  list<T> tail = {}; // pseudo list to quickly access last element and determine if the first list already was copied.
algorithm
  for l in inLst loop
    (outLst, tail) := listAppendTail(outLst, tail, l);
  end for;
end test_appendListTail;

Using ordinary listAppend() like the following should not suffer from this slow-down:

for ... loop
  finalLst := listAppend(nextLst, finalLst);
end for;

[Willi Braun]

Some more improvements for the SimCode phase are done in ​PR490.

SimCode timings for ScalableTestSuite.Mechanical.Strings.ScaledExperiments.StringModelica:

N	without PR490	with PR490
4	0.395906757	0.277792809
8	0.764668169	0.410150604
16	1.876711165	0.714030386
32	6.107274449	1.600489924
64	24.21463418	3.889687883

and for TestSuite.Electrical.TransmissionLine.ScaledExperiments.TransmissionLineModelica:

N	without PR490	with PR490
160	0.59180795	0.548860394
320	1.93300644	1.144715092
640	4.154565855	2.731017459
1280	13.942011015	6.910039283

It seems still not linear, but I suspect this might connect to the evaluation of start-values, where we use the module evalFunc and there exist an other open ticket #3553.

[Francesco Casella]

These are indeed excellent results!

I have started a ​Hudson job with the ScalableTestSuite library to get a more comprehensive view of the effects of the recent commits - results should be soon available ​here. Let's see what we get.

[Willi Braun]

I think this job is based on a fresh run of LINUX_NIGHTLY_BUILD, so let's wait a bit for the newest version, that contains also ​the changes from PR490.

[Adrian Pop]

This job is based on job OpenModelica_TEST_ALL_LIBRARIES which builds a new OMC and runs the coverage testing so you only get one new OMC each night at 2:00. I think we should change it so it builds the new OMC. I'll look into it.

[Adrian Pop]

I now changed LibraryExperimental job to *always* build the newest OMC and then run the coverage with it.

[Francesco Casella]

Thanks! I always forgot that this job would use the latest nightly, but this is now much better. On-demand jobs like this should not depend on others.

[Martin Sjölund]

Milestone pushed to 1.9.5

[Martin Sjölund]

Milestone renamed

[Francesco Casella]

The situation has further improved now, and the scaling is now only slightly superlinear, so I'm closing this ticket.

[Martin Sjölund]

Milestone deleted