FMUs exported from OM are very heavy

[jean-philippe.tavella@…]

As I found a workaround for ticket #6262 I now try to test some FMUs exported from OM, especially for multi-FMU co-initialization with our co-simulator DNG (​https://bitbucket.org/simulage/daccosim/wiki/Home).

Positive feedback: OM FMUs seem to correctly run and being in conformance with FMI 2.0 for co-simulation. Doing a graph with 2 FMUs iterative coinit is correct and results are identical to those given from Dymola FMUs.
Negative feedback: the size of OM FMUs is much more important comparing to Dymola FMUs (with a factor of 10 for the .dll file).
Is there any solution to reduce that? Will be an issue with heavy graphs containing lots of FMUs...
I attach for comparison an OM FMU and a Dymola FMU from the same model.

[jean-philippe.tavella@…]

Dymola FMU

[Francesco Casella]

I suspect that those .dll's have lots of stuff in common. Would it be possible to factor it out and have one runtime multi-threaded dll that works for all the FMUs, so that they could only have model-specific code?

[Martin Sjölund]

The OM FMU for Windows is large because it includes lapack (which is not available in the base Windows 10 install, and this we must include it in the FMU).
For Linux, we generate an FMU that dynamically links lapack (it's so common that we assume everyone has it installed), and the FMU is only 915 kB (smaller than the Dymola FMU even though OM also includes all the source code).

It should also be possible to use the configure script to compile the full source FMU and not link the runtime since we have a smaller version that only includes select portions of lapack. Like the file I am attaching below (it should give the same results as the larger one and the dll is smaller than Dymola's).

[jean-philippe.tavella@…]

Replying to sjoelund.se:

The OM FMU for Windows is large because it includes lapack (which is not available in the base Windows 10 install, and this we must include it in the FMU).
For Linux, we generate an FMU that dynamically links lapack (it's so common that we assume everyone has it installed), and the FMU is only 915 kB (smaller than the Dymola FMU even though OM also includes all the source code).

It should also be possible to use the configure script to compile the full source FMU and not link the runtime since we have a smaller version that only includes select portions of lapack. Like the file I am attaching below (it should give the same results as the larger one and the dll is smaller than Dymola's).

Sorry but the given MVPartCrossCompiled.fmu is for ME and unhappily I can only use/test CS FMUs. Could you send me a light CS FMU for testing? And tell me how to generate it from OMEdit?

Another idea: is not it a possibility to do as for Linux (dynamic Lapack link) and give users the process to install their Lapack for Windows if possible?

[Martin Sjölund]

Another idea: is not it a possibility to do as for Linux (dynamic Lapack link) and give users the process to install their Lapack for Windows if possible?

No, and it won't be compatible with other versions of lapack. If you want a small FMU on Windows, use WSL. (And it's against the FMI specification to depend on dll's not in the OS)

I'll attach a light CS FMU, but you will need to use Linux to generate it. Or a mos-script in Windows I guess. Unless your OMEdit has a button for x86_64-w64-mingw32 compilation. I don't have Windows, so I cannot check that :)

[Martin Sjölund]

ME+CS small FMU

[jean-philippe.tavella@…]

Replying to sjoelund.se:

Another idea: is not it a possibility to do as for Linux (dynamic Lapack link) and give users the process to install their Lapack for Windows if possible?

No, and it won't be compatible with other versions of lapack. If you want a small FMU on Windows, use WSL. (And it's against the FMI specification to depend on dll's not in the OS)

I'll attach a light CS FMU, but you will need to use Linux to generate it. Or a mos-script in Windows I guess. Unless your OMEdit has a button for x86_64-w64-mingw32 compilation. I don't have Windows, so I cannot check that :)

I sucessfully tested your light FMU with our cosimulator DACCOSIM NG.
For the process to generate it, I would prefer not having to install tools like WSL (not so easy on EDF PCs). Is there a possibility for people on your side to explicit a simple and robust process to generate them from Windows?

[Francesco Casella]

Replying to sjoelund.se:

The OM FMU for Windows is large because it includes lapack (which is not available in the base Windows 10 install, and this we must include it in the FMU).
For Linux, we generate an FMU that dynamically links lapack (it's so common that we assume everyone has it installed), and the FMU is only 915 kB (smaller than the Dymola FMU even though OM also includes all the source code).

I understand we need a very limited portion of lapack in most cases (basically just the linear dense solver and any auxiliary functions it may need), and some FMUs may not even need that, if the CS algorithm is explicit and there are no algebraic loops in the model.

In order to generate lightweight FMU dlls, what about compiling lapack from sources and statically linking only the actually used functions?

[Martin Sjölund]

In order to generate lightweight FMU dlls, what about compiling lapack from sources and statically linking only the actually used functions?

That's what we do for source-code FMUs. But Windows has problems running the configure scripts for that.

Maybe we'd need a static-from-source target or something that deals with this for Windows...

[Martin Sjölund]

It might also be possible to generate the static FMU runtime with only this subset of lapack and remove lapack linking in the default target.

[Martin Sjölund]

So I think this actually works if we just stop linking lapack with the FMU. However, many MSL models use lapack as well and it's not going to link easily by just passing -llapack. So we have this code always linking lapack because at least then everything works.

To start getting the size down, I suggest one of the Windows devs tries the following script:

loadModel(Modelica);
loadString("model M
  Real r = der(r);
  Real anorm;
algorithm
  anorm := Modelica.Math.Matrices.LAPACK.dlange(A=[r;r]);
end M;");
buildModelFMU(M, fmuType="me_cs", platforms={"static"});
getErrorString();

And start removing linker commands from:
​https://github.com/OpenModelica/OpenModelica/blob/master/OMCompiler/Compiler/Util/Autoconf.mo.omdev.mingw#L30

Until it stops working. Because when I cross-compile I need pretty much nothing. So I think only the things that lapack needs should remain there...

And maybe we could add a flag for how to link lapack on Windows and simply not link lapack by default...

[Francesco Casella]

Replying to sjoelund.se:

So I think this actually works if we just stop linking lapack with the FMU. However, many MSL models use lapack as well

That was the reason behind my proposal to compile lapack from source and link it statically. The linker would then figure out which functions are needed automatically, right?

So we have this code always linking lapack because at least then everything works.

Premature optimization is the root of all evil. But in this case, we do have a need for optimization, if you want to have hundreds of FMUs loaded in memory.

[Martin Sjölund]

That was the reason behind my proposal to compile lapack from source and link it statically. The linker would then figure out which functions are needed automatically, right?

Wrong. It links everything. It's already linking the whole lapack statically, and that's the problem

[jean-philippe.tavella@…]

I would like to find a user-friendly solution.
I tried this kind of script file:
loadModel(Modelica);
loadFile("tankBarrel.mo");
buildModelFMU(tankBarrel.System4parts, fmuType="cs", platforms={"static"});
deleteFile("tankBarrel_System4parts_FMU.log");
deleteFile("tankBarrel_System4parts_FMU.libs");
deleteFile("tankBarrel_System4parts_FMU.makefile");
getErrorString();

and I'm happy with the result as the FMU is pretty light.
I can then manually still reduce the size by removing unnecessary files:

• the .lib and .libs file in binaries/win64
• the .log file at the root
• the .log file in resources

Only keeping the .dll, the .json and the .xml file is enough for perfectly running...

An option somewhere to get this kind of reduced .fmu file would be great. Either with scriptfile or via the GUI.

[Francesco Casella]

Replying to sjoelund.se:

That was the reason behind my proposal to compile lapack from source and link it statically. The linker would then figure out which functions are needed automatically, right?

Wrong. It links everything. It's already linking the whole lapack statically, and that's the problem

Aha. Is that a bug or a feature?

I quote from ​Wikipedia

With static linking, it is enough to include those parts of the library that are directly and indirectly referenced by the target executable (or target library). With dynamic libraries, the entire library is loaded, as it is not known in advance which functions will be invoked by applications. Whether this advantage is significant in practice depends on the structure of the library.

[Martin Sjölund]

Replying to casella:

Aha. Is that a bug or a feature?

Feature. In C++ you cannot remove static initializers. For C you can do it, but you increase linking speeds by a lot. The flags needed for that aren't universal either. Perhaps it could be done for Windows specifically, but I am not sure it's supported on Windows binaries.

[Martin Sjölund]

Replying to jean-philippe.tavella@…:

An option somewhere to get this kind of reduced .fmu file would be great. Either with scriptfile or via the GUI.

​https://www.openmodelica.org/doc/OpenModelicaUsersGuide/latest/omchelptext.html#omcflag-fmifilter

If you choose to create a black box FMU, some log-files are omitted from what I could tell in the source code. I'm unsure if the lib files are removed though.

[jean-philippe.tavella@…]

Replying to sjoelund.se:

​https://www.openmodelica.org/doc/OpenModelicaUsersGuide/latest/omchelptext.html#omcflag-fmifilter

The blackbox option is not suitable for me as it hides everything except inputs and outputs.

I think an option is missing (both in omc and in OMEdit) to remove:

• the .lib and .libs file in binaries/win64
• the .log file at the root
• the .log file in resources

[Francesco Casella]

Replying to sjoelund.se:

Replying to casella:

Aha. Is that a bug or a feature?

Feature. In C++ you cannot remove static initializers. For C you can do it, but you increase linking speeds by a lot. The flags needed for that aren't universal either. Perhaps it could be done for Windows specifically, but I am not sure it's supported on Windows binaries.

Maybe a flag to enable a stripped-down version of lapack that works most of the time is the best compromise.

[Francesco Casella]

Retargeted to 1.18.0 because of 1.17.0 timed release.

[Francesco Casella]

Ticket retargeted after milestone closed