Opened 5 years ago
Closed 3 years ago
#5726 closed enhancement (fixed)
Categorize open-source libraries in OpenModelica for different uses
| Reported by: | Francesco Casella | Owned by: | Francesco Casella |
|---|---|---|---|
| Priority: | critical | Milestone: | 1.19.0 |
| Component: | Third-Party Libraries | Version: | |
| Keywords: | Cc: | Dietmar Winkler |
Description (last modified by )
During the Board meeting of 2 Dec 2019, we decided to classify the available open-source Modelica libraries (see e.g. https://github.com/modelica-3rdparty/) into four categories:
- Production-grade libraries, full or nearly full OMC support
- Production-grade libraries, partial OMC support, commitment to get to full support ASAP
- Libraries under development or lacking proper maintenance (a.k.a. "experimental")
- Obsolete or otherwise unsupported libraries
The first category includes all production-grade libraries which are actively maintained by their authors, which are responsive to feedback from OSMC in case of issues such as use of illegal Modelica code that hampers the compilation by OMC, and where 100% or nearly 100% available library tests pass successfully.
The second category is the same, except that the ratio of tests passing with OMC is still not close to 100%, but we have a commitment to eventually get there, and we state it explicitly.
With the goal of promoting OMC, we should publish reports on the web that only include models from those two sets, declaring explicitly that we are working to improve the coverage in the case of set no. 2.
The goal of OSMC regarding these two categories is to get to 100% green lights everywhere.
The third category is interesting for developers, which may find it helpful to get continuous testing of their libraries with OpenModelica, and also interesting for the Consortium to promote and increase the use of OMC among new library developers. For example, it could include master/HEAD revision of libraries whose latest release(s) belong to sets 1 and 2, or libraries under early development such as OpenHPL.
It also provides useful information for people who may consider picking up the development of open-source libraries whose development has been discontinued by their original authors, e.g. ElectricalEnergyStorage.
However, any failure in this category is not OSMC's responsibility, so we should prepare separate web reports for these libraries, whose purpose is to help their development, not to judge on the quality of implementation of OMC.
The fourth and last category includes libraries that are obsolete, because they have been superseded by more recent ones. For example, the Annex60 library is no longer developed, because it has been ported and merged into the IBPSA library, so it makes no sense to still refer to it, paricularly if there are any broken models. I would also consider very old versions of production-grade libraries in this set (e.g. Buildings prior to 5.1.0), since they most likely contain bugs that may lower the test passing ratio, but will never be fixed because the libraries have been superseded by more recent versions.
It also includes libraries that we may decide not to support because they are not conforming to the Modelica standard and their authors are not responding to pull requests or other kind of request for proper fixes.
This classification will provide the basis for web reports. It also provides the basis for the choice of libraries that we ship with the OMC installer, and that will be managed in the future by the Modelica package manager, which will only belong to sets 1. and 2., since 3. is best managed directly using a GIT tool, and 4. should never been considered in the first place.
We should probably provide some feedback in the installation procedure regarding the level of support, by linking each library of categories 1. and 2. to the corresponding web reports.
For some libraries, it seems that the authors have continued the development for a while without having a proper release. If the master version is clearly better than the latest released version, we are provisionally including that as the reference one; in the meantime, we are contacting all authors to make sure that the latest improvements are included in an official release.
I am tentatively scheduling this task for 1.15.0.
1. Production-grade libraries, full OMC support
2. Production-grade libraries, partial OMC support
3. Development, experimental, no longer maintained libraries
| Library | Version | URL |
|---|---|---|
| Buildings | latest | https://github.com/lbl-srg/modelica-buildings |
| ExternData | latest | https://github.com/modelica-3rdparty/ExternData |
| IBPSA | latest | https://github.com/ibpsa/modelica-ibpsa |
| IDEAS | master | https://github.com/open-ideas/IDEAS |
| ModelicaTest | latest | https://github.com/modelica/ModelicaStandardLibrary/tree/master/ModelicaTest |
| Modelica_DeviceDrivers | latest | https://github.com/modelica-3rdparty/Modelica_DeviceDrivers |
| Modelica | master | https://github.com/modelica/ModelicaStandardLibrary |
| OpenHPL | master | https://github.com/simulatino/OpenHPL |
| PhotoVoltaics | master | https://github.com/christiankral/PhotoVoltaics |
| PhotoVoltaics_TGM | master | https://github.com/christiankral/PhotoVoltaics |
| Physiolibrary | master | https://github.com/MarekMatejak/Physiolibrary/ |
| Physiomodel | master | https://github.com/physiology/Physiomodel |
| PlanarMechanics | master | https://github.com/dzimmer/PlanarMechanics |
| PowerSystems | master | https://github.com/modelica-3rdparty/PowerSystems |
4. Obsolete or discontinued
| Library | Version | Report | URL | Reason |
|---|---|---|---|---|
| ConPNLib | master | https://github.com/lochel/ConPNlib/commits/master | Superseded by PNLib | |
| ElectricalEnergyStorage | latest | https://github.com/modelica-3rdparty/ElectricalEnergyStorage | Discontinued by the authors | |
| FCSys | 0.2.6 | https://kdavies4.github.io/FCSys/ Development | Currently obsolete (old MSL dependency), development could resume in 2020 | |
| IdealizedContact | N/A | https://github.com/tbeu/IdealizedContact | No longer maintained | |
| ModelicaTest | 3.2.1 | https://github.com/modelica/ModelicaStandardLibrary/tree/maint/3.2.1/ModelicaTest | Superseded by MSL 3.2.2, no longer maintained | |
| Modelica_Noise | 1.0 Beta 1 | https://github.com/DLR-SR/Noise | Obsolete, merged in to MSL 3.2.2 | |
| Modelica_StateGraph2 | 2.0.2 | https://github.com/HansOlsson/Modelica_StateGraph2 | Not fully Modelica compliant | |
| ObjectStab | 1.0.3 | https://github.com/modelica-3rdparty/ObjectStab | Marked as discontinued on GitHub | |
| OpenHydraulics | master | https://github.com/cparedis/OpenHydraulics/commits/master | No longer maintained by the authors | |
| SiemensPower | master | ? | No longer maintained by the authors |
Change History (40)
comment:1 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:2 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:3 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:4 by , 5 years ago
| Description: | modified (diff) |
|---|
follow-up: 6 comment:5 by , 5 years ago
comment:6 by , 5 years ago
Replying to dietmarw:
MSL 3.2.2 is also no longer maintained.
That is true, but there are still many libraries out there that have a annotation(uses(Modelica(version="3.2.2"))); annotation, so if we want to support them we need to include 3.2.2 in the "good libraries" list for a while. I think keeping the last and one but last version active is a reasonable compromise (we were asked to do so with Buildings, for example).
I am contacting library owners to make sure they update the uses annotation, but that would take some time.
How would you suggest to proceed?
comment:7 by , 5 years ago
| Description: | modified (diff) |
|---|
follow-up: 10 comment:8 by , 5 years ago
Since MSL 3.2.3 is backward compatible the user libraries could be loaded with 3.2.3 silently (just warning about it) without having to update the uses annotation. Good luck with getting the upstream to update the libs ;-)
comment:9 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:10 by , 5 years ago
Replying to dietmarw:
Since MSL 3.2.3 is backward compatible the user libraries could be loaded with 3.2.3 silently (just warning about it) without having to update the uses annotation.
This is probably the right way to go, but I suspect there are different opinions in the community and this is not really standardized. Maybe it's time to build a consensus on what is the right way to manage this. I will work on that within MAP-LIB.
Good luck with getting the upstream to update the libs ;-)
Thank you!
comment:11 by , 5 years ago
@dietmarw, I've updated the description with an explanation of the rationale of choosing master vs. latest release as the reference implementation. I hope you are fine with that.
follow-up: 13 comment:12 by , 5 years ago
I think FCSys should go into set 4) - it hasn't seen a commit since 2014, and does not even claim compatibility with OM.
Also, I would like to propose a refinement to formulation:
Coverage, afaik from my software testing experience, refers to the fraction of source code that your test suite covers/executes. I.e., "100% coverage" means 100% of your source code is covered by tests.
What I think you mean with "100% coverage" is that 100% of your test suite passes, which is a completely different thing, and should not be confused.
An illustrative example: You have a package with 10k lines of code, but only 5 tests that cover a single model. All your tests pass down to the reference results (so "100% green" as you say above), but your "[code|test] coverage" is probably in the range of 1%.
comment:13 by , 5 years ago
Replying to Christoph Buchner <buchner@…>:
I think FCSys should go into set 4) - it hasn't seen a commit since 2014, and does not even claim compatibility with OM.
The library seems well done, so I am trying to contact the authors to see if they are interested in an update. Hence the TBD comments. I'll move it to Cat 4 if I don't get feedback anytime soon.
Regarding compatibility with OM, of course it's not mentioned, as running this library in 2014 with omc would have been science fiction :)
My personal position is that a Modelica library should first and foremost be Modelica compliant. There are maybe a few gray areas, since the only formal part of the specification is the grammar, while the semantics is given by plain English text, which is subject to interpretation and ambiguity. However, in many cases libraries contain clearly illegal code, which is accepted by other tools. This is a big problem in terms of conflict of interest.
One one side, selfish tool vendors may wonder, why do I need to bug my users by making previously functional models fail compilation, just because of Modelica compliance, which could make my customers more easily move to the competition in the future?
On the other side, if you sell a Modelica tool or a Modelica library, you are benefitting from community work and are favoured by the Modelica ecosystem, so it's your interest to keep that healthy and growing.
In general, I believe there is a consensus that tools should be stricter, though the transition may take time. OpenModelica has the goal to become the golden standard in term of Modelica Specification compliance, of course it's still some way to go before we get there, but I think we should be firm in trying not to lure in users by accepting illegal Modelica code that they run in other tools. Of course there are cases where the Modelica specification is really bad, and then the idea is to change the specification with the due process. If there is consensus on Modelica changes, we can of course test implement them and accept the code even if the specification is still not officially approved.
That said, there's plenty of excellent libraries out there which are 100% Modelica compliant and don't work with OMC, which are currently placed in Cat. 2. Our goal is to make OMC support them, not to have their authors find workarounds to make them work with OMC.
Also, I would like to propose a refinement to formulation:
Coverage, afaik from my software testing experience, refers to the fraction of source code that your test suite covers/executes. I.e., "100% coverage" means 100% of your source code is covered by tests.
What I think you mean with "100% coverage" is that 100% of your test suite passes, which is a completely different thing, and should not be confused.
An illustrative example: You have a package with 10k lines of code, but only 5 tests that cover a single model. All your tests pass down to the reference results (so "100% green" as you say above), but your "[code|test] coverage" is probably in the range of 1%.
Sure. When we say 100% coverage we mean 100% library coverage, i.e., we simulate 100% of executable models with the experiment(StopTime) annotation in a library successfully. Mind you, it's not our testsuite, it's the entirety of a library contents.
Of course that doesn't mean that the tool is perfect. It just means that all publicly available models within that library work as expected, i.e. they simulate and, in case we are provided with reference results, they produce the same results, within numerical tolerances.
Is "100% library coverage" fine for you, or what would you prefer?
follow-up: 15 comment:14 by , 5 years ago
Of course that doesn't mean that the tool is perfect. It just means that all publicly available models within that library work as expected, i.e. they simulate and, in case we are provided with reference results, they produce the same results, within numerical tolerances.
If I understand correctly, you'd typically have an experiment annotation in library examples, right?
If so, then effectively the set of library examples comprise the "test suite".
Then we are back in the situation where, if there are only few examples in a given library, which execute (successfully) only a part of the library, we have low "coverage" (in software engineering lingo), but a "fully passing test suite" (again, SWE lingo).
As an example, you surely have seen "build passing" badges in Readmes of projects on Github, e.g. here.
The tests that are typically run in a Modelica context, where the simulation output is compared to a "known-good" reference results, are typically called regression tests.
Where the whole comparison stops being fully clear is that, in my experience, you normally don't merge PRs where the test suite fails any tests (that are not known as flaky/failing) - you either fix your code or the test suite. Of course, in this situation, the test suite is actually out of your hands, so you can't fix that without going upstream to the library author.
However, giving an indication of how much of their tests are passing is of course very useful!
Is "100% library coverage" fine for you, or what would you prefer?
"100% (library) tests passing" is the most acurate thing I could come up with.
comment:15 by , 5 years ago
Replying to Christoph Buchner <buchner@…>:
If I understand correctly, you'd typically have an
experimentannotation in library examples, right?
Yes. We may actually consider to skip some of them in some situations. For example, if you take the ScalableTestSuite report, there are 29/263 tests failing. However, this is due to the insufficient resource that are allocated on the test server for each test run. All these tests run fine on my server, given enough juice, except the ones mentioned in #4845. In this case it makes sense to exclude some tests, giving a good motivation on why they are not included.
If so, then effectively the set of library examples comprise the "test suite".
Yes
Then we are back in the situation where, if there are only few examples in a given library, which execute (successfully) only a part of the library, we have low "coverage" (in software engineering lingo), but a "fully passing test suite" (again, SWE lingo).
OK.
The tests that are typically run in a Modelica context, where the simulation output is compared to a "known-good" reference results, are typically called regression tests.
I'm aware of this. In this case I understand the main focus is to catch commits that break previously successful tests (a.k.a. regressions)
Where the whole comparison stops being fully clear is that, in my experience, you normally don't merge PRs where the test suite fails any tests
This is actually the case for the "core" OpenModelica testsuite. Developers file pull requests, which can only be merged if there are no regressions.
However, we cannot run the full testuite on each commit because it simply takes too much time. Also, it may depend on the library code, which is outside our control. We run the full testsuite a few times a day and report regressions here:
https://libraries.openmodelica.org/branches/history/
e.g. the regressions for the newInst job, using the new frontend, is here
https://libraries.openmodelica.org/branches/history/newInst/
These are inspected manually and if necessary the appropriate course of action is taken, e.g. fixing bugs in omc, contacting external library developers, or filing PR's for the external libraries.
Is "100% library coverage" fine for you, or what would you prefer?
"100% (library) tests passing" is the most acurate thing I could come up with.
Excellent, I'll use that from now on.
Thanks!
comment:16 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:17 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:18 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:19 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:20 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:21 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:22 by , 5 years ago
| Description: | modified (diff) |
|---|
First list complete. Some libraries are undergoing a release process after some years with just a few minor fixes, will need to update that accordingly.
comment:23 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:24 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:25 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:26 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:27 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:28 by , 5 years ago
It seems to me that on mac OM still installs unsupported libraries. Here follows the full list.
For instance we see buildings 1.4, 1.5, 1.6, ect. this makes installation through macports huge in time and size.
Is this intentional?
omlib-adgenkinetics
omlib-admsl
omlib-advancednoise-1.0.1
omlib-aixlib-0.4.0
omlib-aixlib-0.9.1
omlib-algebratestsuite
omlib-annex60-1.0.0
omlib-approxspline-1.0.0
omlib-arduino-0.1.0
omlib-biochem-1.0.1
omlib-bondgraph
omlib-bondlib-2.3
omlib-brineprop-0.5.5
omlib-buildingcontrollib-0.1.0
omlib-buildings-1.4
omlib-buildings-1.5
omlib-buildings-1.6
omlib-buildings-2.0.0
omlib-buildings-2.1.0
omlib-buildings-3.0.0
omlib-buildings-4.0.0
omlib-buildings-5.0.1
omlib-buildings-5.1.0
omlib-buildings-6.0.0
omlib-buildings-latest
omlib-buildingsystems-2.0.0-beta
omlib-buildsyspro-3.3.0
omlib-chemical-1.2.0
omlib-complex-3.2.1
omlib-complex-3.2.2
omlib-complex-3.2.3
omlib-complex-trunk
omlib-complexlib
omlib-conpnlib
omlib-deploystructlib
omlib-deslib-1.6.1
omlib-disheatlib-1.2
omlib-drivecontrol-3.1.0
omlib-electricalenergystorage-3.2.2
omlib-electromechanicaldrives-2.2.0
omlib-emoth-1.4.0
omlib-extendedpetrinets-1.0
omlib-externalmemorylib
omlib-externdata-2.5.0
omlib-failuremodes-1.2.1
omlib-fastbuildings-0.0
omlib-faulttriggering-0.6.6
omlib-fcsys-0.2.6
omlib-fcsystest
omlib-feeddrivelibrary
omlib-flight
omlib-fmitest
omlib-fractionalorder
omlib-fuzzycontrol
omlib-gnu-scientificlibrary
omlib-greenhouses
omlib-hansermodelica-1.1.0
omlib-helmholtzmedia
omlib-ibpsa-3.0.0
omlib-ibpsa-latest
omlib-idealizedcontact-0.2.0
omlib-industrialcontrolsystems-1.1.0
omlib-instantaneoussymmetricalcomponents
omlib-ipsl-1.1.0
omlib-keywordio-0.9.0
omlib-libras
omlib-linearmpc-1
omlib-manualtracking
omlib-messagepack-0.1.1
omlib-modelica-1.6
omlib-modelica-2.2.2
omlib-modelica-3.1
omlib-modelica-3.2.2
omlib-modelica-3.2.3
omlib-modelica-devicedrivers-1.8.0
omlib-modelica-linearsystems2-2.3.5
omlib-modelica-noise-1.0-beta.1
omlib-modelica-requirements-0.6
omlib-modelica-stategraph2-2.0.3
omlib-modelica-synchronous-0.93.0
omlib-modelica-trunk
omlib-modelicaadditions-1.5
omlib-modelicaads
omlib-modelicabyexample-0.5.0
omlib-modelicacompliance-3.2
omlib-modelicadevs-1
omlib-modelicareference
omlib-modelicareference-trunk
omlib-modelicaservices-1.0
omlib-modelicaservices-3.2.1
omlib-modelicaservices-3.2.2
omlib-modelicaservices-3.2.3
omlib-modelicaservices-trunk
omlib-modelicatest-3.2.1
omlib-modelicatest-3.2.2
omlib-modelicatest-3.2.3
omlib-modelicatest-trunk
omlib-modelicatestoverdetermined-3.2.2
omlib-modpowersystems
omlib-motorcycledynamics
omlib-mvemlib-1.0.1
omlib-ncdatareader2-2.5.0
omlib-neuralnetwork-1.0
omlib-nuclear
omlib-objectstab-1.1-dev
omlib-obsoletemodelica3
omlib-openbldc
omlib-openfdm
omlib-openhpl-1.1.1
omlib-openhydraulics-1.0
omlib-openipsl-2.0.0-dev
omlib-optimisers-0.1
omlib-photovoltaics-1.5.0
omlib-photovoltaics-tgm-1.5.0
omlib-physiolibrary-2.3.2-beta
omlib-physiomodel-1.0.1-beta
omlib-planarmechanics-1.4.1
omlib-planarmechanicstest
omlib-pnlib-2.2
omlib-powerflow-0.3
omlib-powersystems-1.0.0
omlib-powersystems-latest
omlib-praxissimulationstechnik
omlib-pvsystems-0.6.3
omlib-qssfluidflow-1.0
omlib-realtimecoordinationlibrary-1.0.4
omlib-scalabletestsuite-1.11.4
omlib-servomechanisms-1.0
omlib-siemenspower-2.1-beta
omlib-siemenspower-2.2
omlib-siemenspower-omctest
omlib-smps
omlib-solartherm-0.2
omlib-soltermica
omlib-spot-0.706.1
omlib-spotexamples-0.706.1
omlib-streamconnectors
omlib-systemdynamics-2.1.1
omlib-test
omlib-testderivative
omlib-thermalseparation-0.2
omlib-thermopower-3.1
omlib-thermosyspro-3.1
omlib-thermosyspro-3.2
omlib-thermosysprologo
omlib-vehicleinterfaces-1.2.2
omlib-vehicleinterfaces-1.2.4
omlib-vvdrlib
omlib-wastewater-2.1.0
omlib-wavelet
omlib-wbehptlib
omlib-wbehvpkg
comment:29 by , 5 years ago
@ceraolo, the actions of this ticket still need to be implemented.
Our current top priority is to release 1.15.0 with replaceable support, this will come next.
comment:31 by , 5 years ago
There are two parts to this:
- Further break up the reports into supported and non-supported lists (manually done). Some libraries are tested with C++/FMI runtime, etc that do not have the same level of support, too.
- OMEdit/etc needs to be able to categorize the libraries (and perhaps a tooltip saying why it goes in this category). This should just need us to create a package.omc-category file or something that we can look for (similar to the package.encoding we put on some old libraries).
Note: The MacOS installer includes all libraries because the realistic alternative is no library and manually downloading the ones you prefer. In the long term, we want a package manager instead (so you can choose any library and any tagged version to download for best flexibility and lowest download size).
comment:32 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:33 by , 5 years ago
| Description: | modified (diff) |
|---|
comment:34 by , 5 years ago
For the record, I add the links to
- the config files for the library testing
- the libraries shipped with OpenModelica (Windows installer)
- the Jenkins testing script
comment:35 by , 4 years ago
| Milestone: | 1.15.0 → 1.16.0 |
|---|
Release 1.15.0 was scrapped, because replaceable support eventually turned out to be more easily implemented in 1.16.0. Hence, all 1.15.0 tickets are rescheduled to 1.16.0
comment:37 by , 4 years ago
| Milestone: | 1.17.0 → 1.18.0 |
|---|
Retargeted to 1.18.0 because of 1.17.0 timed release.
comment:39 by , 3 years ago
| Milestone: | → 1.19.0 |
|---|---|
| Status: | new → assigned |
Discussion continues on GitHub https://github.com/OpenModelica/OpenModelica/issues/5726
comment:40 by , 3 years ago
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
MSL 3.2.2 is also no longer maintained.