Support for disabling Jacobians

[Martin Sjölund]

If an explicit solver is used, we do not need the information in the Jacobian. For large models, it is desirable to be able to disable generation of Jacobians on a flag.

[Lennart Ochel]

Do you mean something like --postOptModules-=detectJacobianSparsePattern? That's already available.

[Francesco Casella]

Not only that. All the generation of code to compute the Jacobian (whether or not it is sparse or coloured) should be disabled, in the back-end, but also in the SimCode and C-compilation. It takes a lot of time and memory which is completly wasted in this case. And it gets worse with increasing model size.

[Lennart Ochel]

Where get those Jacobians calculated? Inside the SimCode phase?

[Willi Braun]

By default we generate _only_ for linear systems symbolic jacobians, what actually is matrix A (in A*x=b), which is needed to get a direct solution of such linear system.
Is this one meant by this ticket?

Otherwise we generate the sparsity information and colouring of a system by default which is not used by explicit solver and therefore the solution from Lennart works.

[Francesco Casella]

Just to make sure I get it correctly:

• the original system (after index reduction) is an index one DAE: F(x, dx/dt, v, t) = 0
• causalization brings it to dx/dt = f(x,t)
• the Jacobian df/dx is computed analytically by default only if f(x,t) is linear w.r.t x
• otherwise, it is computed by numerical differentiation, using colouring to reduce the calls to f(x+dx,t) to a minimum

Is this correct?

[Willi Braun]

No, the Jacobian df/dx is computed analytically f(x,t) w.r.t. x only with the flag --generateSymbolicJacobian is set.

[Martin Sjölund]

Using --postOptModules-=detectJacobianSparsePattern, there still seems to be some Jacobians generated:

Notification: Performance of postOpt calculateStrongComponentJacobians (initialization): time 0.04305/3122, GC stats:
Notification: Performance of postOpt calculateStrongComponentJacobians (simulation): time 0.003566/3161, GC stats:
Notification: Performance of postOpt calculateStateSetsJacobians (simulation): time 3.074e-05/3161, GC stats:
Notification: Performance of SimCode generated analytical Jacobians: time 116.6/3279, GC stats:

[Lennart Ochel]

Replying to sjoelund.se:

Notification: Performance of postOpt calculateStrongComponentJacobians (initialization): time 0.04305/3122, GC stats:

This can be disabled using --initOptModules-=calculateStrongComponentJacobians.

Notification: Performance of postOpt calculateStrongComponentJacobians (simulation): time 0.003566/3161, GC stats:

This can be disabled using --postOptModules-=calculateStrongComponentJacobians.

Notification: Performance of postOpt calculateStateSetsJacobians (simulation): time 3.074e-05/3161, GC stats:

This can be disabled using --postOptModules-=calculateStateSetsJacobians.

However, I'm not sure if everything still works fine if these modules are disabled.

[Martin Sjölund]

Well, those 3 are fast. What does "SimCode generated analytical Jacobians" actually do? Is it important? :)

[Lennart Ochel]

Replying to sjoelund.se:

Well, those 3 are fast. What does "SimCode generated analytical Jacobians" actually do? Is it important? :)

This should disable all the symbolic Jacobians, which means that also the SimCode task "generated analytical Jacobians" should be performed in no-time since there is nothing to do, right?

[Lennart Ochel]

Replying to wbraun:

No, the Jacobian df/dx is computed analytically f(x,t) w.r.t. x only with the flag --generateSymbolicJacobian is set.

BTW: Better use --postOptModules+=generateSymbolicJacobian.

[Willi Braun]

Replying to sjoelund.se:

Using --postOptModules-=detectJacobianSparsePattern, there still seems to be some Jacobians generated:

[...]
Notification: Performance of SimCode generated analytical Jacobians: time 116.6/3279, GC stats:

Hm, my guess is we measure there something different that run while the SimCode phase, since there is really nothing mandatory to do in the function where this execStat is executed.

Replying to lochel:

Replying to sjoelund.se:

Notification: Performance of postOpt calculateStrongComponentJacobians (initialization): time 0.04305/3122, GC stats:

This can be disabled using --initOptModules-=calculateStrongComponentJacobians.

Notification: Performance of postOpt calculateStrongComponentJacobians (simulation): time 0.003566/3161, GC stats:

This can be disabled using --postOptModules-=calculateStrongComponentJacobians.

Notification: Performance of postOpt calculateStateSetsJacobians (simulation): time 3.074e-05/3161, GC stats:

This can be disabled using --postOptModules-=calculateStateSetsJacobians.

However, I'm not sure if everything still works fine if these modules are disabled.

But these modules are more or less necessary to perform a simulation run.
If one would deactivate --postOptModules-=calculateStateSetsJacobians there is no alternative and a simulation run with dynamic state selection will fail.
Also the module calculateStrongComponentJacobians, which basically calculates for a linear torn system the jacobian matrix (again, that's actually A, of A*x=b), then the system will be not solvable, since we have no alternative in place.

[Francesco Casella]

Sorry, let me focus back on the core issue: we have a test case in which we get:

Notification: Performance of SimCode generated analytical Jacobians: time 116.6/3279, GC stats:

It is not 100% clear which Jacobians does this notification message refer to. The test in question has a (very) large linear strong component, so it might be the case that this time is spent doing what Willi explained in comment:5. Willi, can you please confirm?

If this is the case, then it should not be switched off. However, it should definitely be made more efficient!

[Francesco Casella]

Please note that in this case we know a priori that the system is index 1 and a fixed choice of states is just fine.

Can we possibly cut down the back-end time by setting stateSelect.always on the system states?

[Willi Braun]

Replying to casella:

Sorry, let me focus back on the core issue: we have a test case in which we get:

Notification: Performance of SimCode generated analytical Jacobians: time 116.6/3279, GC stats:

It is not 100% clear which Jacobians does this notification message refer to. The test in question has a (very) large linear strong component, so it might be the case that this time is spent doing what Willi explained in comment:5. Willi, can you please confirm?

If this is the case, then it should not be switched off. However, it should definitely be made more efficient!

The point is that we use here execStat, what internal is a tick tock time measurements and in the whole simCode phase this is the only execStat. So that time is not meaningful at all, since it contains a lot of different things which are performed from the last tick in the Backend phase to this first tock in simCode phase.
I've just added ​https://github.com/wibraun/OMCompiler/tree/fixSimCodeExecStat some more significant time measurement statements in the simCode phase.
My guess is that we lose the time while we creating modelInfo and all the variables.

Replying to casella:

Please note that in this case we know a priori that the system is index 1 and a fixed choice of states is just fine.

Can we possibly cut down the back-end time by setting stateSelect.always on the system states?

If the system has clear states and the index reduction has no issue to detect that, than I think we don't lose any time on dynamic state selection at all, so I guess that is not the issue here.

[Francesco Casella]

OK. Given the information stated so far, I can close the ticket

[Lennart Ochel]

I'm going to set resolution to invalid, since there was nothing to fix.

[Francesco Casella]

Thanks for pointing this out :)

[Martin Sjölund]

Milestone renamed

[Martin Sjölund]

Milestone renamed