The front-end takes too much time doing basic operations

[casella]

Consider the attached test package. The model M contains N scalar differential equations. That's it: no models within models, no connection equations, no inheritance, no functions, no package constants, no fancy stuff.

These are the execution time of the front-end on my Windows pc:

N	Time [s]
2000	5.2
4000	20.5
6000	45.1
8000	63.5

I don't understand why the time grows almost quadratically with the number of equations. Also, looking at the last row and doing the math, I don't understand why instantiating one scalar equation with four variables should take a whooping 8 ms time - you can do a lot of things in 8 ms on a I5 processor @ 2.5 GHz...

It seems to me that there is something fundamentally wrong in how this thing is handled, which makes it impossible to deal with models with more than 10000 states (which is not such a big deal, we're not taling about finite element or 3D fluid dynamics models) in a reasonable amount of time.

I hope this example is small enough to allow a detailed analysis :)

[casella]

Adrian pointed out that the problem might be the extends clause that is inefficient. I prepared other test cases that do not use extends, but I get the same results. See the updated attachment.

[perost]

I just finished running some profiling on the largest model, and one issue I've found is that we expand the dimensions to a list of expressions when looking up a cref. And since we also expand the for-loop this blows up when you have large arrays. I will see if it's possible to remedy this somehow.

[perost]

Improved in f45a8b2, the front end now scales slightly better than linear. Here are some measurements on my computer (which flattened the 8000 model in ~33 sec before the fix):

N	Time [s]	Equations/s
2000	1.1	1829
4000	2.0	1963
6000	3.0	2006
8000	3.8	2100
100000	44.3	2259

Still pretty bad, but I can't see anything obvious when profiling anymore. Improving the performance in any significant way probably means redesigning the way we do instantiation. A lot of the time is spent doing loop unrolling though, i.e. instantiating the loop body for each iteration. It might be possible to instantiate the loop body once with a non-constant iterator and then creating the equations by replacing the iterator in each equation. I'm not sure if that's possible without breaking things though, but I will investigate.

[rfranke]

This was a good move!

[casella]

See also #3678

[casella]

I have added to the ScalableTestSuite a model (CascadedFirstOrder_N_XXX) similar to the one attached to this ticket, save that I made each equation depending on the previous one, in order to avoid the somewhat irrealistic setup of having N totally independent subsystems. This way, we can keep the progress on this front monitored from Hudson.

I noticed that the front end allocates about 66 kB of memory for each variable. This figure is consistent across a range from 200 to 25600 equations (two orders of magnitude). BTW, this figure is similar to what I get with my large network models.

I think it should be possible to investigate a bit more where this memory is going, as this model is very simple in structure, has no connections, no inner-outer, no replaceable, and no other fancy Modelica features - just one extends an an array of simple ODE equations.

What do you think?

[casella]

I confirm the very rough figure of 2000 equations/s also in our much larger test cases. I guess this is really too slow to handle BIG models, but this is probably something to consider for the new instantiation, so I'm closing this ticket.