Opened 6 years ago
Last modified 3 years ago
#4992 assigned task
Should we skip some versions of the Buildings library in Hudson testing and in the OMC installer?
| Reported by: | Francesco Casella | Owned by: | Martin Sjölund |
|---|---|---|---|
| Priority: | high | Milestone: | |
| Component: | Testing Framework | Version: | |
| Keywords: | Cc: | Adrian Pop, Michael Wetter |
Description
@mwetter, if you check https://libraries.openmodelica.org/branches/master/ you can see there are currently 7 versions of the Buildings library, plus 2 versions of the older Annex60 library, in the OMC testsuite, each containing from 300 to 900 test models.
Every day, each of these libraries is tested with the latest 7 stable releases of OMC plus the master development version (see https://libraries.openmodelica.org/branches/overview-combined.html), as well as with the latest development version and the fmi, daeMode, newInst and daeMode-newInst flags, see https://libraries.openmodelica.org/branches/
That's really a lot of testing work, which is sometimes straining our servers a bit. It is of course of the utmost importance for us to test the latest versions of the library, so we can check the progress of coverage and performance of OMC on them. I am not sure if there is some added value at also checking all the previous versions, on a daily basis, ao we need your input about this.
In fact, this issue also affects the size of the OMC installer and the installation time, since all those versions are included.
I don't know if old versions of these libraries are still actively used and thus have value for end-users, or rather if they now have mainly a historical value. Can you please add a comment about which versions you'd like to see in the daily reports and in the installer, so we can streamline both processes? Please note that libraries using versions of the Modelica Standard Library older than 3.2.1 (which had several portability issues) cannot really be supported seriously by OMC, unless they are upgraded to use MSL 3.2.1 (or later).
I also take the opportunity of showing the recent progress with the new front end. Coverage of Buildings is finally taking off, and we expect it to reach 100% in the following few months, hopefully overcoming all the outstanding issues that could not be fixed in the old front end.
Thanks
Change History (22)
comment:1 by , 6 years ago
comment:2 by , 6 years ago
| Cc: | added |
|---|---|
| Owner: | changed from to |
| Status: | new → assigned |
| Type: | discussion → task |
Excellent!
@sjolund.se, @adrpo, can you please remove
Annex60
Annex60 1.0.0
Buildings
Buildings 2.0.0
Buildings 2.1.0
Buildings 3.0.0
Buildings 5.0.0
Buildings 5.0.1
from the list of libraries in the testsuite and in the set provided with the installer, and add
Buildings 5.1.0 https://github.com/lbl-srg/modelica-buildings/releases/tag/v5.1.0
IBPSA 2.0.0 https://github.com/ibpsa/modelica-ibpsa/releases/tag/v2.0.0
I guess this will significantly trim the Windows installer and speed up all the testsuite jobs.
Thank you!
comment:3 by , 6 years ago
I discussed this with @sjoelund.se. He sees a value at seeing what happens with older libraries as well, so we agreed to move the older libraries to a job that only runs on weekends.
As to the installer, the situation is different in Linux and Windows. Linux has a library manager, so having more libraries available is no big deal. Windows eventually also should, but currently all libraries are downloaded and (re)installed every time a new version is installed, even if they haven't changed.
I'm not sure how much overhead the extra Buildings libraries are, if you don't install them. Probably not much, as I guess the library code is compressed somehow. It only really takes time to write the libraries to the disk because they are spread over so many files, but one can always choose which ones to install.
In fact, the real strategic question is: should we actually install libraries that we know in advance have a poor coverage ratio? This would be irritating for most new users, because if you are proposed to install a library, you'd assume it just works, except that you later find out it doesn't, which is kind of odd.
If we have enough faith, this issue should resolve itself automatically with v. 2.0.0, as we expect a drastically improved convergence ratio on most libraries. Otherwise, I think we should really reconsider which libraries we ship with OMC, and at least make a distinction between those which are (mostly) supported and those which still aren't.
comment:5 by , 5 years ago
@sjolund.se, @adrpo: Buildings 6.0.0 has been released:
https://github.com/lbl-srg/modelica-buildings/releases
Can you please add it to the OpenModelica distribution.
From https://libraries.openmodelica.org/branches/master/ I see that you may want to remove 2.0.0, or maybe remove all except "latest", 6.0.0 and 5.1.0.
Also, Annex60 and Annex60_1.0.0 can in my view be removed, and instead use IBPSA "latest", which is already present, and add 3.0.0. The repo is https://github.com/ibpsa/modelica-ibpsa/releases
comment:6 by , 5 years ago
I'm also strongly in favor of removing old libraries. They are no longer maintained, and they may also contain non-standard Modelica code that we'll never support. I think we should add more libraries to the testsuite, rather than keeping a lot of versions of old libraries.
follow-up: 8 comment:7 by , 5 years ago
Currently, these libraries are never installed on Linux/OSX. You select which libraries to install and it makes sense to keep the old ones for people who have other libraries that depend on them.
What needs to change is the Windows installer including too many libraries by default.
My proposal for that was to get a master thesis to create an amazing Modelica package manager (something like an improved version of impact and with OMEdit / OMC integration). And here every single tagged release of all Modelica libraries would be optionally available.
As for https://libraries.openmodelica.org/branches/master/ it includes old directories that are not updated. We only run Buildings 5.0.1 and latest today.
comment:8 by , 5 years ago
Replying to sjoelund.se:
My proposal for that was to get a master thesis to create an amazing Modelica package manager (something like an improved version of
impactand with OMEdit / OMC integration). And here every single tagged release of all Modelica libraries would be optionally available.
Having a Modelica package manager would be nice. That we use it to manage old libraries that use older versions of the MSL won't be very useful for OMC, because MSL < 3.2.1 wasn't even conforming to the standard, and even MSL 3.2.1 and 3.2.2 had a lot of issues (e.g. missing each) that won't work with the new front-end.
As for https://libraries.openmodelica.org/branches/master/ it includes old directories that are not updated. We only run Buildings 5.0.1 and latest today.
I find this a bit confusing. Shouldn't we remove old directories that are not updated and only keep the ones that are?
comment:9 by , 5 years ago
@adrpo, @sjoelund.se, could you please take care of the following issues, following Michael's advice in comment:5:
- remove the Buildings, Buildings_1.6, Buildings_2.0.0, Buildings_2.1.0, Buildings_3.0.0
directories from https://libraries.openmodelica.org/branches/master/. They refer to very old versions that we're not testing since 2017/2018, so I think they unnecessarily clutter the summary page - if we have enough space we could still keep them for the record, but then they should be moved to an
obsoleteorOlddirectory - update the Jenkins scripts for Buildings so we only test 5.1.0 (which supersedes 5.0.1), 6.0.0 (which is the latest stable release) and the latest one on master (which we already have)
- remove Annex60 from the Jenkins library testsuite (it is superseded by IBPSA)
- remove all versions of Buildings prior to 5.1.0 from the Windows installer, as well as Annex60, as they are now considered obsolete, so we can trim the installer a little bit.
Focusing too much on older version of these libraries is completely pointless, as we don't cover them anyway. Let's try to get the latest ones, which are maintained properly, to work asap with rev. 2.0.0
comment:10 by , 5 years ago
| Milestone: | 1.14.0 → 1.16.0 |
|---|
Releasing 1.14.0 which is stable and has many improvements w.r.t. 1.13.2. This issue is rescheduled to 1.16.0
comment:11 by , 5 years ago
In addition to the issues in comment:9, we should add the reference results from the following sources
comment:13 by , 4 years ago
The Buildings library 7.0.0 has been released, https://github.com/lbl-srg/modelica-buildings/releases
Could you please add this to the libraries that are used in the OpenModelica regression tests.
comment:14 by , 4 years ago
Yes, I planned to overhaul completely the list of tested library, see #5726. I just need to find some time to do that, sorry.
comment:15 by , 4 years ago
| Cc: | added |
|---|
We are testing Buildings 5.0.1 and Buildings latest.
Should I switch 5.0.1 to 7.0.0 instead?
comment:16 by , 4 years ago
5.0.1 is obsolete and should have been replaced by 5.1.0. I guess we should have at least 6.0.0, 7.0.0 and latest (i.e. master/HEAD). @mwetter, are you still interested in the testing of 5.1.0 or can we skip that?
comment:17 by , 4 years ago
I suggest you use the master/HEAD plus 7.0.0 and maybe 6.0.0 which is about a year old. I don't see it worthwhile to go back further.
comment:19 by , 4 years ago
Is done since yesterday. All the tests done today should include these:
https://libraries.openmodelica.org/branches/overview-newinst.html
comment:21 by , 4 years ago
| Milestone: | 1.17.0 → 1.18.0 |
|---|
Retargeted to 1.18.0 because of 1.17.0 timed release.
@casella: I am very happy to see the increased coverage, and I am sure there will be many new users if the fluid and building models work as I get questions about OpenModelica coverage weekly.
As to the regression testing, I suggest to use "Buildings_latest" and the highest number of the release (currently 5.1.0 since last week). Once we get good coverage and hence an OpenModelica/Buildings user base, we could think about keeping latest releases of each non-backward compatible series (highest number in the 5.x series, 6.x series etc.) I see little value in going earlier than 5.x
Annex60 could be removed as this has been renamed to IBPSA. Again, "IBPSA_latest" and the highest release number starting at 2.x would be great to include.