Handling libraries in 1.17.0

[casella]

We plan to support both MSL 3.2.3 and 4.0.0 (the latter with some limitations on Clocked components) in 1.17.0, see #6274.

This requires to completely re-think the way we handle system (and non-system) libraries in OMEdit. Possibly in a way which can then be easily integrated with the package manager, see #6020.

I made some reflections on the topic. It is clear to me that there are five main use cases in OMEdit

1. Work with an existing model or package that only uses system libraries, managed via MODELICAPATH
2. Work with an existing model or package that (also) uses libraries outside MODELICAPATH, e.g., development versions of packages, including the MSL
3. Start a new model or package from scratch, using the Modelica Standard Library only
4. Start a new model or package from scratch, using some specific system library, that inevitably uses MSL
5. Start a new model or package from scratch using some custom user library outside MODELICAPATH

First of all, when OMEdit is loaded, I would start with a blank session. No defaults. People will often work with different projects, some using MSL 3.2.3, some using 4.0.0, there is no reason why one should choose a default.

Case 1. is very easily handled: you open your existing model or package, that includes uses annotations, and those are used by OMC to load the corresponding libraries from the MODELICAPATH. By default, that covers the system directories for libraries, there should be an easy way to edit MODELICAPATH from the GUI, if one wants to add other directories.

Regarding uses annotations, I understand OMEdit creates them automatically as soon as some components from a library are dragged and dropped into a package, so this is going to work smoothly without doing anything special.

Case 2. can be covered if one tweaks the MODELICAPATH, putting custom directories ahead of system ones. Another good way to cover it is to allow people to run .mos scripts that will load libraries from well-specified paths. These scripts could be written manually or, in the future, automatically by OMEdit upon request, and executed from the File Menu and/or from a suitable button in the toolbar.

I believe Case 3. and 4. should be handled by presenting the user with a window that includes a list of installed libraries (MSL first, then other system libraries). One can then click on the required libraries and load them. Then one creates his own model or package, starts dragging and dropping stuff, and that creates the uses annotation automatically. Next time, you just open your model or package and you are back to Case 1. In version 1.18.0, this window will be more tightly integrated with the package manager, and we'll also tackle the issue of upgrading libraries to newer versions of their dependencies.

Basically, this window replaces the File | System Libraries menu, which is too limited, as it does not include different versions of the same library, in particular of the MSL, which is instead essential. I would also split it in two parts, one for the MSL and another one for all the other libraries, given the very special status of the MSL, that should not be downloaded, but comes shipped with the tool.

The only potentially critical issue in Case 4. is that if you select libraries that have dependencies on different versions of the same library, e.g., one system library that uses MSL 3.2.3 and one that uses MLS 4.0.0, chaos would ensue, particularly since we don't yet have conversion scripts. We should have some means to make sure that you can only make selections that make sense.

My suggestion to handle the case when at some point there is a uses annotation that wants to load a library that was already loaded with an incompatible version, (e.g. first I load a library that uses MSL 3.2.3, and then another that uses MSL 4.0.0) is to give a clear *error* message (not a warning) and unload everything. The current status (i.e. we continue anyway with a bland warning) is not good, because it leads to situations where nothing works properly. This is in particular true now that we have MSL 3.2.3 and MSL 4.0.0: currently OMEdit says that we may have problems if the two are incompatible, but we know for sure they are incompatible, and basically nothing at all will work. Better to stay safe and unload everything.

I'm a bit unsure about Case 5. I guess we could add custom libraries with specific file system locations to a section of the installed libraries window, and load them from there. The only problem I see in this case is that if I have multiple versions of library A in different locations in the file system, and multiple versions of library B that uses A in different locations in the file system, when I load a specific version of B, the uses annotation in it will automatically load a version of A that may not be the one I am interested into, and once it's loaded, it won't be unloaded if another version of A is loaded next.

This issue can be fixed by loading the custom libraries in forward order, i.e. first the appropriate MSL, then the indicated version of A, then the indicated version of B. But this is difficult to explain and to handle in general. Also, to replicate Case 5 in the future, you would need to save a .mos file with the appropriate locations.

Maybe for the time being we should suggest users falling under Case 5, which can be classified as advanced, to always use scripts to load libraries at the beginning of the session, and to do so in forward order, to avoid the automatic uses annotation to override their choice. An experienced user should understand that. After all, that's what I've been doing in Dymola for almost 20 years, and it always worked well.

Last, but not least, we should refer users to a man page explaining these concepts the first time they install 1.17.0. The concept should be natural and easy to follow, but we have to make it very explicit the first time.

[casella]

On a second thought, implementing this proposal is probably incompatible with the timing of the 1.17.0 release, which is already late. We are way past feature freeze with that version. Also, we need to have it tested a bit by alpha users before we release it.

I think the best option for 1.17.0 is to keep 3.2.3 and 4.0.0 in the installer as we have it now, and then have the installer modify the options in the library window so that version 3.2.3 of the MSL (not default) is loaded. This makes sure that the default behaviour is unchanged w.r.t. 1.16.x. We shouldn't harass users with a radical change of paradigm in 1.17.0, that we may be forced to change again when implementing conversion scripts in 1.18.0

For those (probably not too many) users that want to use MSL 4.0.0 already in 1.17.x, we write instructions in the release notes that if you also want to use 4.0.0 you can remove the MSL from the list of automatically loaded libraries in OMEdit, and rely on the uses annotation to get the right library to be loaded. This is how I have been using 1.18.0 recently, and it's not bad.

The original proposal of this ticket can then be integrated with the conversion scripts and the package manager in 1.18.0.

[casella]

Update after today's webmeeting: the problem with the installer doing some stuff is that on Linux there is no installer.

Proposed solution: upon starting a newly installed OMEdit 1.17.0, a screen pops up with the following text

Setup of Modelica Standard Library version.

OpenModelica 1.17.x supports both Modelica Standard Library (MSL) v. 3.2.3 and v. 4.0.0. Please note that synchronous components in Modelica.Clocked are still not fully reliable, while most other models work fine in both versions.

MSL 3.2.3 and 4.0.0 are mutually incompatible, because of changes of class names and paths; for example, Modelica.SIunits became Modelica.Units.SI in version 4.0.0; see ​link for further information. Please note that conversion scripts are not yet available in OpenModelica 1.17.x, so you need to use other Modelica tools to upgrade existing libraries to use MSL 4.0.0. Conversion scripts support are planned to be supported in version 1.18.0.

On Windows, both version of the MSL are installed automatically by the installer. On Linux, you have to install them using apt-get or the package manager, as explained in the installation instructions.

You have three options:

1. Automatically load MSL 3.2.3 in the package browser when starting OMEdit. You can then load other models or packages that use MSL 3.2.3, or start new ones that will use it. If you then open a model or package that uses MSL 4.0.0, errors will occur. This option is recommended if you are not interested in MSL 4.0.0 and you would like to get the same behaviour as in OpenModelica 1.16.x.
2. Automatically load MSL 4.0.0 in the package browser when starting OMEdit. You can then load other models or packages that use MSL 4.0.0, or start new ones that will use it. If you then open a model or package that uses MSL 3.2.3, errors will occur. This option is recommended if you exclusively use new libraries depending on MSL 4.0.0.
3. Do not load MSL when starting OMEdit. When you open a model or library, the appropriate version of MSL will be loaded automatically, based on the uses() annotation of library being opened. This option is recommended if you work with different projects, some using MSL 3.2.3 and some others using MSL 4.0.0. It is also recommended if you are a developer of the Modelica Standard Library, so you want to load your own modified version instead of the pre-installed, officially released read-only version.

Please choose one option:
( ) Load MSL 3.2.3 when starting OMEdit
( ) Load MSL 4.0.0 when starting OMEdit
( ) Do not load MSL when starting OMEdit

You can later change this setting by going to Tools | Options | Libraries, where you can add or remove the Modelica library from the list of automatically loaded system libraries, as well as specify which version of the library you want to load. Version tag "default" will load the latest released version (i.e. 4.0.0 for MSL).

The OK button should be activated only when an option is chosen, and the libraries set accordingly.

I guess this thing should only be shown once when 1.17.0 is first run. Later on you can change the settings with the Libraries menu, as explained in the text.

[casella]

See ​PR 7211 for master and ​PR 7216 for maintenance/1.17

[casella]

1.17.0-beta.2 for Windows does not include MSL 4.0.0 in the installer.

[adrpo]

We needed to also update the libraries as by default MSL 4.0.0 was not created in build/lib/omlibrary. PR ​https://github.com/OpenModelica/OpenModelica/pull/7244 should fix that.

[adeas31]

PR ​https://github.com/OpenModelica/OpenModelicaSetup/pull/16 fixes the installer to include MSL 4.0.0

[adrpo]

Changed version to 1.17.0-dev.beta3:
​https://github.com/OpenModelica/OpenModelicaSetup/commit/ddb1d9ee20c744760cc2ce2a4e3d5a85ea5220e0
when PR ​https://github.com/OpenModelica/OpenModelica/pull/7244 is done, I will tag it and start a new Window build.

[adrpo]

OM v1.17.0-dev.beta3 is building now. Will be available in a couple of hours here:
​https://build.openmodelica.org/omc/builds/windows/releases/1.17/maintenance/
then I will move it to is final place here:
​https://build.openmodelica.org/omc/builds/windows/releases/1.17/dev.beta3/

[adrpo]

Of course, the first build only worked for 32bit, then it ran out of disc space for the 64bit but didn't stop so it pushed a broken install. Then I made some logic to exit when the installer fails to build but the logic was wrong so all builds have failed last night. I changed the logic this morning, hopefully to a better one and restarted the 1.17.0-dev.beta3 :)

[casella]

Still running: ​https://test.openmodelica.org/jenkins/job/Windows/job/OM_WIN_RELEASE_1.17/8/

[casella]

Tested successfully in 1.17.0-dev.beta3.

[phannebohm]

On small to medium sized screens the pop up window is too small to hold all the text and it gets chopped off before the radio buttons.

[casella]

@adeas31, can you please add a scrollbar to this window? Please remember to push on master and port on maintenance/1.17

[adeas31]

PR for it is already there and is just waiting a review from Philip. See ​https://github.com/OpenModelica/OpenModelica/pull/7279

[adeas31]

@phannebohm the PR is merged. Please test it in the nightly build and give feedback.

[phannebohm]

Works fine!