64bit OMC for Windows

[Christian Schubert]

Hi,

this is just a ticket so that we don't forget. Adrian and I made a working 64bit version of OMC for Windows which resides in the branch mingw64.

Before we can merge it into the trunk, a few decisions have to be made:

• Shall we distribute both a 32 and a 64bit OMC simultaneously
• If so, we need a new folder structure, i.e.
  • bin/omc.exe and bin/omc64.exe, or
  • bin/omc.exe and bin64/omc.exe
  same holds true for lib/
• What about OMDEV: Shall we have one combined 32bit and 64bit repo or an OMDEV32 and an OMDEV64 repo

Opinions or further questions are highly appreciated.

[anonymous]

It would be nice to rebuild the CMake build system instead of the OMDev environment. This makes it easy to create various versions for different complier and operating systems. mingw is not my favourite Compiler for Windows.

[Martin Sjölund]

Better would be to use the autoconf system we use in all the other places since cmake sucks.

[Adrian Pop]

We already have CMakeLists.txt for almost everything but nobody to keep them updated.
If somebody wishes to take care of that and keep them updated I'm all for it but be prepared to work quite a bit because is not easy to update those files. The dependencies of OpenModelica are huge and some of them do not have any CMake like build systems.

[Adrian Pop]

BTW, there is already a branch for this:
https://trac.openmodelica.org/OpenModelica/browser/branches/mingw64
but is a bit outdated and needs to be merged with the changes from trunk.

[anonymous]

If you build on windows autoconf sucks, too.

[Martin Sjölund]

Then build Windows binaries on Linux. It compiles faster ;)

[Christian Schubert]

I also think that the current way of building omc is a bit convoluted as we use three different build systems:

• autoconf on linux
• plain makefiles on windows
• some CMake for dependencies and runtimes

I agree with the above:
While I like the features of autoconf, its support for windows is exceptionally bad.

@Martin: What is the problem with CMake in your opinion?

[Martin Sjölund]

cmake cannot do most of the things autoconf does. As soon as there is no cmake module for the library you want to link, you will do things wrong. It's been many months since I reported some issues with the FMIL cmake files on BSD, but they failed to fix them thus far.

Most dependencies of OpenModelica lack cmake projects. The only reason they work on Windows is that they are pre-compiled in OMDev. Remove OMDev, compile everything from scratch and see if you think cmake is still a good idea.

With makefiles you can do pretty much anything. Look into Compiler/boot/Makefile.common and see if you can replicate the omc bootstrapping process in cmake for example.

autoconf works fine on Windows from what I can tell. The plain makefiles use it for dependencies like libModelicaExternalC after all. That is, as long as you use the mingw target - which is what we build...

[Stephan]

Have a look at the Ogre project (​https://bitbucket.org/sinbad/ogre/). They use CMake in a very comfortable way. They don’t distribute dependency sources. You have to take care about the dependencies by yourself. If you like, you can download and build the dependencies in a separate package (for windows ​http://www.ogre3d.org/forums/viewtopic.php?f=4&t=69274) or a separate repository (​https://bitbucket.org/cabalistic/ogredeps) with the sources and created CMake files. Yes, it is a lot of work to create CMake files for dependencies. The main project contains only the core code and is fully configured by CMake. Tests and examples are also done by CMake. You could build Ogre for Windows, Linux, OSX, Android, iOS, WindowsRT and Windows Phone without any creepy environment workaround like mingw. You can use mingw if you like but you don’t have to use it. Also it is not necessary to create a new branch for 64 bit or any other special configuration. Removing the dependencies from the core projects removes a lot of work keeping them up to date. Using mingw at Windows is like using wine at Linux. Keep in mind that autoconf is an “old” tool (last version 2.69 24-Apr-2012) and only available for Linux. CMake is still under active development (last version 2.8.12.2 28-Jan-2014).

We (Rexroth) would prefer CMake to build OpenModelica for various systems and hardware with their native compilers.

„cmake cannot do most of the things autoconf does”
@Martin: Could you please provide some examples?

Btw: ​http://stackoverflow.com/questions/4071880/autotools-vs-cmake-vs-scons

[Adrian Pop]

The dependencies that we have *are* needed and
is impossible to do without them.

You cannot compile OpenModelica without them.
Or if you can, you get generated code that does
not compile due to missing dependencies.

I personally have nothing against using CMake,
but we don't have a CMake expert yet and no
time to change the build system at the moment
as we're focusing on more important things.

If you want to keep the CMake build of OpenModelica
up to date is fine with us.

[Martin Sjölund]

The problem with cmake is that it is under active development! Many projects depend on the latest and greatest cmake being installed. Autoconf depends on a single developer somewhere has a copy of autotools installed. This is a huge difference. We build OpenModelica on Linux distributions around 4 years old. These distributions will not ship with an up-to-date cmake and the build system has to work for these as well (cmake 2.8 is not an option for example).

By contrast, autoconf works fine. Not only on Linux, but also on BSD and OSX. Most of the cmake dependencies we had (Modelica3D, FMIL, etc) had to be patched because otherwise they cannot find headers in default paths on the OS.

Also, we really do not want people to have to compile all the omc dependencies by hand because most of them are not standard packages in OS'es. Many numerical solvers are uncommon to have as packages. We had to create our own sundials package on OSX for example.

But really, have a look at source:trunk/Compiler/boot/Makefile.common and figure out how to do that in CMake. Note that you need a 3-phase (IIRC) approach because the files that you should compile are unknown a priori and may change during the bootstrapping process.

[Stephan]

Replying to adrpo:

The dependencies that we have *are* needed and
is impossible to do without them.

You cannot compile OpenModelica without them.
Or if you can, you get generated code that does
not compile due to missing dependencies.

I can't compile Ogre without dependencies, too. But Ogre doesn't distribute the dependencies in their core repository. CMake is searching for them. If you like you can compile them by yourself. If you don't like to compile them you can use a precompiled package. CMake is searching for binary files not for the source code. You have to keep the dependencies separate from the main project. Build the main project with CMake and everybody is happy because it’s working on all platform. If you like to build the dependencies with autoconf feel free to do that.

Replying to sjoelund.se:

The problem with cmake is that it is under active development! Many projects depend on the latest and greatest cmake being installed. Autoconf depends on a single developer somewhere has a copy of autotools installed. This is a huge difference. We build OpenModelica on Linux distributions around 4 years old. These distributions will not ship with an up-to-date cmake and the build system has to work for these as well (cmake 2.8 is not an option for example).

You don't have to set on 2.8. You can use 2.6 if you want. Which system are you using? I like to test that.

By contrast, autoconf works fine. Not only on Linux, but also on BSD and OSX. Most of the cmake dependencies we had (Modelica3D, FMIL, etc) had to be patched because otherwise they cannot find headers in default paths on the OS.

OSX is based on BSD. So autoconf works on Linux and BSD but not on Windows. You might be a Linux wizard but our customers are using Windows. Also we are developing on Windows. Ok, I understand the problem. You have to patch the dependencies to get it working on various platforms. Why not coding the patchs in CMake? This would decrease the work on updating the dependencies.

[Adrian Pop]

We don't distribute the dependencies in our core repository either.
The dependencies are using SVN *external*.
OMDev package is also external.

I don't think the build system is the problem here.

The problem is that all the code (including core
dependencies) needs to be portable across a lot
of operating systems (Windows, Linux, MacOS)
architectures (32 and 64 bit) and compilers
(GCC, CLANG, Visual Studio). CMake does not
help at all with regards to code portability,
is just another build system. The only thing
you gain by using CMake is project files and
makefiles for Visual Studio, nothing more.

The customers can use Windows, they don't
build OpenModelica by themselves I guess,
they get an installer.

[Martin Sjölund]

Replying to Stephan:

I can't compile Ogre without dependencies, too. But Ogre doesn't distribute the dependencies in their core repository. CMake is searching for them. If you like you can compile them by yourself.

No, you can't. If you use an Ubuntu-compiled version of libgc, it will not work because we assume different configure flags. You have to use exactly the versions we are using. Same thing for the antlr parser, etc.

You don't have to set on 2.8. You can use 2.6 if you want. Which system are you using? I like to test that.

Debian oldstable still works as far as I know.

OSX is based on BSD. So autoconf works on Linux and BSD but not on Windows. You might be a Linux wizard but our customers are using Windows.

autoconf works on way more platforms than that though ;) Pretty much anything gcc runs on (and since we almost depend on gcc or clang, that's good).

Windows is a bit of a problem. OpenModelica pretty much assumes you are using a POSIX system. You need a command-line environment and support much of the POSIX library for best performance (POSIX threads, mmap, etc). The Windows support is mostly a hack at this point. I wouldn't be surprised if a Virtual Machine running Ubuntu completed the testsuite faster than running it directly in Windows...

Note that autoconf does run on Windows (MinGW). It can also support 64-bit code. If someone with a bit of time and a Windows machine wanted to, we could have support for both MinGW32, MinGW64, OSX, BSD, and Linux in the same autoconf. It really is not that hard. I am certain it would be less work than writing stupid cmake files and making everyone set env.variables DEBUG=1 VERBOSE=0 in their bashrc before they had went insane from cmake hiding all useful output by default.

Also, if you want to work on CMake stuff, please fix SimulationRuntime/cpp. It re-compiles object-files every time despite no changes being made to the tree. It's really annoying, and since I don't know cmake, I have no time to fix it.

[Stephan Schweig]

Replying to adrpo:

The problem is that all the code (including core dependencies) needs to be portable across a lot of operating systems (Windows, Linux, MacOS) architectures (32 and 64 bit) and compilers (GCC, CLANG, Visual Studio).

Yes, the intension of OM is to run on Linux. Other platforms are not interesting. This is a shame, because for example, modern game engine is now being designed for multiple platforms. Also apps for smartphones.

Replying to adrpo:

CMake does not help at all with regards to code portability, is just another build system. The only thing you gain by using CMake is project files and makefiles for Visual Studio, nothing more.

True, but this is very comfortable to work with. I can debug everything with the used Visual Studio or Intel Studio. I'm not able to debug the runtime build with gcc inside of Visual Studio. I just get an error that it's not working. Also, CMake produces make files for various compiler and platforms. You don't have to setup a whole tool chain for each platform and configuration. Of course you have to collect all different compiler flags one time to set them up in CMake.

Replying to sjoelund.se:

Windows is a bit of a problem. OpenModelica pretty much assumes you are using a POSIX system. You need a command-line environment and support much of the POSIX library for best performance (POSIX threads, mmap, etc). The Windows support is mostly a hack at this point. I wouldn't be surprised if a Virtual Machine running Ubuntu completed the testsuite faster than running it directly in Windows...

This is not a surprise! A Windows application like MS Office is running faster on native Windows than in Wine, too. The problem is that OM and the build system has been developed for Linux only. Btw, there are a lot of OpenSource libraries for cross platform development e.g. TBB, Boost or Poco for threads and boost for mmap. Most of the POSIX stuff could be replaced by boost.

Replying to sjoelund.se:

Note that autoconf does run on Windows (MinGW). It can also support 64-bit code. If someone with a bit of time and a Windows machine wanted to, we could have support for both MinGW32, MinGW64, OSX, BSD, and Linux in the same autoconf. It really is not that hard. I am certain it would be less work than writing stupid cmake files and making everyone set env.variables DEBUG=1 VERBOSE=0 in their bashrc before they had went insane from cmake hiding all useful output by default.

I could use similar arguments for CMake. Note that CMake runs on most operating systems. It can also support 64-bit in the same CMake file without large modifications. It really is not that hard. I’m certain it would be less work using CMake than using this stupid tool chain of OMDev and Eclipse on Windows. Btw, you still get the output of gcc etc. because CMake just configure your make files – you still build your code with make & gcc on Linux.

Replying to sjoelund.se:

Also, if you want to work on CMake stuff, please fix SimulationRuntime/cpp. It re-compiles object-files every time despite no changes being made to the tree. It's really annoying, and since I don't know cmake, I have no time to fix it.

I was told that this behavior is by design.

[Martin Sjölund]

Replying to stephan:

I can debug everything with the used Visual Studio or Intel Studio. I'm not able to debug the runtime build with gcc inside of Visual Studio. I just get an error that it's not working.

Sounds like you are having issues due to too much C++ being used in OpenModelica. Let's rip that out :)

A Windows application like MS Office is running faster on native Windows than in Wine, too.

First of all, Wine is not an emulator or a virtual machine. Games run faster on Linux in Wine than they do in native Windows.
The point was a VM like VirtualBox running Linux will run faster than the same code native in Windows.

The problem is that OM and the build system has been developed for Linux only. Btw, there are a lot of OpenSource libraries for cross platform development e.g. TBB, Boost or Poco for threads and boost for mmap. Most of the POSIX stuff could be replaced by boost.

No, you cannot substitute in boost. It is a C++ library and not portable. Having a C++ dependency is pretty insane when trying to write portable code. The C++ ABI changes a lot, so you cannot mix compilers. The C++ parts we currently have in the runtime is causing us headaches every day.

You need an ANSI C cross-platform library. MinGW does some things. We could of course use glib for some additional abstraction. But then why did we start using Qt for the GUI? :)

I could use similar arguments for CMake. Note that CMake runs on most operating systems. It can also support 64-bit in the same CMake file without large modifications.

Our autoconf works fine in 32/64-bit without any modifications at all.

Also, if you want to work on CMake stuff, please fix SimulationRuntime/cpp. It re-compiles object-files every time despite no changes being made to the tree. It's really annoying, and since I don't know cmake, I have no time to fix it.

I was told that this behavior is by design.

None of the other parts of the build system do this. You should run "make clean" to accomplish this behaviour.

[Martin Sjölund]

This ticket was not closed for 1.9.1, which has now been released. It was batch modified for milestone 1.9.2 (but maybe an empty milestone was more appropriate; feel free to change it).

[anonymous]

I agree to stephan. Let's get rid of that autoconf stuff. Industrial applications must be able to run on Windows. The bloated OMDev toolchain makes it difficult to work with a native Windows compiler (eg. MS Visual Studio) or on low powered systems (Eclipse is a pain). CMake this simplifies enormously. You can put the necessary 3rd party libraries in a git repository and include it by CMake.

[Adrian Pop]

Replying to anonymous:

I agree to stephan. Let's get rid of that autoconf stuff. Industrial applications must be able to run on Windows. The bloated OMDev toolchain makes it difficult to work with a native Windows compiler (eg. MS Visual Studio) or on low powered systems (Eclipse is a pain). CMake this simplifies enormously. You can put the necessary 3rd party libraries in a git repository and include it by CMake.

We cannot get rid of the "autoconf stuff" as Linux/Mac depends on it.
There is no "native" Windows compiler, you'll have to install it (MS Studio Express or whatever) so there is no difference here compared to OMDev (even compared in size). And you don't have to use Eclipse, you can use any editor and compile OpenModelica from the console.

If anybody which is a CMake expert can provide (and also maintain) the CMake files to compile the entire OpenModelica compiler, including the graphical clients, then is fine, we have nothing against it but we don't have time to do it ourselfs.

From my side I had only pain with CMake. Countless times I had to debug what is doing because compilation just failed without any useful message. Still, as I said previously I have nothing against CMake. There is however no silver bullet and whatever tool gets the job done properly is fine with me.

[Adrian Pop]

I'm wondering how people would like to have the 64bit OMC on Windows:

1. As as completely different installation pretty much as we have now for 32 bit but with 64 bit executables and libraries. Basically we'll have 2 separate installers one for 32 bit and one for 64 bit.

2. Combined 32 and 64 bit installer where people can run one or the other as they want. This is a bit more complicated as we need to have bin64 and lib64 in the installer and the clients need a bit of update on which omc to run, etc. Also, the size of the installer will grow about twice as big as it is now.

Please feel free to give your opinion.

[Martin Sjölund]

Replying to adrpo:

If anybody which is a CMake expert can provide (and also maintain) the CMake files to compile the entire OpenModelica compiler, including the graphical clients, then is fine, we have nothing against it but we don't have time to do it ourselfs.

Also note that some of the 3rdParty code does not use CMake, so regardless you need to do things with autotools and rely on those projects having native build systems for win32 as well (both would need to be handled by CMake).

Not to mention someone needs to write a CMake system for MetaModelica projects. A Makefile allows you to add dependencies after you run configure (the cmake command), in the middle of the build process. Dependencies are not static, and in fact they change when you edit MetaModelica files so this is an essential feature that as far as I know is very hard to solve with CMake.

[Stephan Schweig]

Replying to adrpo:

Replying to anonymous:

I agree to stephan. Let's get rid of that autoconf stuff. Industrial applications must be able to run on Windows. The bloated OMDev toolchain makes it difficult to work with a native Windows compiler (eg. MS Visual Studio) or on low powered systems (Eclipse is a pain). CMake this simplifies enormously. You can put the necessary 3rd party libraries in a git repository and include it by CMake.

We cannot get rid of the "autoconf stuff" as Linux/Mac depends on it.
There is no "native" Windows compiler, you'll have to install it (MS Studio Express or whatever) so there is no difference here compared to OMDev (even compared in size). And you don't have to use Eclipse, you can use any editor and compile OpenModelica from the console.

Do you know KDE? KDE is running on Windows, Linux and even on OSX. They moved to CMake in front of version 4 to have a cross platform build environment. Linux and Mac can also work with CMake.

Replying to adrpo:

From my side I had only pain with CMake. Countless times I had to debug what is doing because compilation just failed without any useful message. Still, as I said previously I have nothing against CMake. There is however no silver bullet and whatever tool gets the job done properly is fine with me.

CMake is using an installed compiler on your system, so if there is no useful message ask your compiler. Btw, CMake is making progress and is providing some functions to output error messages. I understand that you don't like CMake, because you've never worked with it.

Replying to sjoelund.se:

Also note that some of the 3rdParty code does not use CMake, so regardless you need to do things with autotools and rely on those projects having native build systems for win32 as well (both would need to be handled by CMake).

Have a look at the OGRE. They are using dependencies without CMake files, but they provide CMake files to build them (​https://bitbucket.org/cabalistic/ogredeps).

[Stephan Schweig]

Replying to adrpo:

I'm wondering how people would like to have the 64bit OMC on Windows:

1. As as completely different installation pretty much as we have now for 32 bit but with 64 bit executables and libraries. Basically we'll have 2 separate installers one for 32 bit and one for 64 bit.

2. Combined 32 and 64 bit installer where people can run one or the other as they want. This is a bit more complicated as we need to have bin64 and lib64 in the installer and the clients need a bit of update on which omc to run, etc. Also, the size of the installer will grow about twice as big as it is now.

Please feel free to give your opinion.

I prefere option 1 if both version can run on the same computer. If you combine Modelica with an other software you maybe need to change between 32 and 64 bit - depending on the software.

[Gerard Lardner]

If you do offer 64-bit OMC then having it as a run-time option in case of conflict with other 32-bit software seems like a good idea.

For industrial users a larger installer is not usually a problem; the company I work for has reasonable bandwidth and no data limits, so size of installer should not be a problem.

[Adrian Pop]

That's the plan. We'll have both 32/64 bit OMC in the installer as that's more flexible and people might have legacy 32 bit libraries.
One should be able to compile 32 bit code from 64 bit OMC and 64 bit code from 32 bit OMC.
I'm working now on the structure of the directories where the various libraries/executables will be.

[Christoph <buchner@…>]

Because an opinion was requested, I think separate 32bit and 64bit installers would be better.

The installer right now is already above 400MB. I quite often update OM with the latest nightly, and am not looking forward to doubling data use (and doubling the download time) for that.
Personally, I haven't felt the necessity of running both 64bit and 32bit OM (and in that case, I don't see why one should not just download two installers, and install consecutively.
Also, I don't know this of course, but maybe openmodelica.org bandwidth and data cost will be (significantly) affected by this?

[Adrian Pop]

Christoph, thank you for your input.
Currently I'm working on having a joined 32 and 64 bit installer but we can also provide separate ones rather easy after that.
Bandwidth for openmodelica.org is not a problem as is supported by the university.

[Martin Sjölund]

Perhaps the installer could be made more modular, updating only packages that actually change. Adrian: perhaps we could host an msys2 repository with OpenModelica packages? So if MSL 3.2.1 changes, only MSL 3.2.1 needs to be upgraded. Of course, with an option to do an offline install as well.

[Adrian Pop]

Having a package for msys2 could be a possibility. I will look into that later on.

[Martin Sjölund]

Not a package. Many packages. One for omc, one for OMEdit, one for OMOptim, one for MSL 3.2.1, one for MSL 3.1, etc :)

We will split the repository at some time anyway.

[Adrian Pop]

Of course. Several packages. I meant it as a suite of packages for OpenModelica.

[Stephan Schweig]

Replying to sjoelund.se:

We will split the repository at some time anyway.

https://trac.openmodelica.org/OpenModelica/ticket/2619
Git with multiple (sub)repositories? Btw, is there any progress on that topic?

[Adrian Pop]

Not yet Stephan. We were busy with a lot of other things but we'll get to it.

[Martin Sjölund]

Moved to new milestone 1.9.4

[Martin Sjölund]

Milestone pushed to 1.9.5

[Martin Sjölund]

Milestone renamed

[Adrian Pop]

This ticket (#2578) and ticket (#3318) should be fixed with a new push to OMDev 165/OpenModelicaExternal/trunk/tools/windows/OMDev (happening now) and PRs:
​https://github.com/OpenModelica/OpenModelica/pull/16
​https://github.com/OpenModelica/OMCompiler/pull/635
​https://github.com/OpenModelica/OMEdit/pull/20
​https://github.com/OpenModelica/OMNotebook/pull/4
​https://github.com/OpenModelica/OMOptim/pull/2
​https://github.com/OpenModelica/OMShell/pull/2
​https://github.com/OpenModelica/OMPlot/pull/2

[Adrian Pop]

We should have a 64 bit version starting today.

[Martin Sjölund]

Milestone deleted