Initialization file not loaded from default directory in OMEdit

[Francesco Casella]

Steps to reproduce:

• Download the attachment of ticket #5824
• Run ImportInitialValues.ResultGenerator. The initial.mat file is generated in the corresponding working directory
• Run ImportInitialValues.ImportModelAnnotationInitialEqs; the model fails because of missing initial.mat file.
• Copy the initial.mat file from the working directory of the first model to the working directory of the second model
• Re-run ImportInitialValues.ImportModelAnnotationInitialEqs. The executable still cannot find the initial.mat file:

unable to read input-file <initial.mat> [No such file or directory]

This is weird, because the command line

d:/temp/OMEdit/ImportInitialValues.ImportModelAnnotationInitialEqs/ImportInitialValues.ImportModelAnnotationInitialEqs.exe -port=51902 -logFormat=xmltcp -override=startTime=0,stopTime=1,stepSize=0.002,tolerance=1e-6,solver=dassl,outputFormat=mat,variableFilter=.* -r=d:/temp/OMEdit/ImportInitialValues.ImportModelAnnotationInitialEqs/ImportInitialValues.ImportModelAnnotationInitialEqs_res.mat -iif=initial.mat -w -lv=LOG_STATS 
-inputPath=d:/temp/OMEdit/ImportInitialValues.ImportModelAnnotationInitialEqs 
-outputPath=d:/temp/OMEdit/ImportInitialValues.ImportModelAnnotationInitialEqs

contains the same input and output paths, so it is not clear why the file is not located there.

Notice that if you put the absolute path of the initial.mat file in the annotation, the import procedure works correctly.

It would be nice to fix this in time for 1.16.0, so the feature described in #5824 is fully functional

Thanks!

[Adeel Asghar]

Maybe the runtime is ignoring the -inputPath flag.

[Andreas Heuermann]

Yes, it looks like it.
I'm trying to fix it.

By the way, I don't think ​-inputPath is the flag we want to change here.
If I change inputPath to temp/OMEdit/ImportInitialValues.ResultGenerator omc tries to find the init file in the wrong location.

 messages = "Simulation execution failed for model: ImportInitialValues.ImportModelAnnotationInitialEqs
assert            | debug   | simulation_input_xml.c: Error: can not read file temp/OMEdit/ImportInitialValues.ResultGenerator/ImportInitialValues.ImportModelAnnotationInitialEqs_init.xml as setup file to the generated simulation code.

I would like to have a annotation like annotation(__OpenModelica_simulationFlags(iif = "../ImportInitialValues.ResultGenerator/initial.mat")); working.

[Andreas Heuermann]

But I can't come up with a mos script to test this without OMEdit (that's quiet complicated to build in WSL and currently broken).

I thought this would behave like in OMEdit but works just fine.
test.mos

// Run ImportInitialValues.ResultGenerator
mkdir("ImportInitialValues.ResultGenerator"); getErrorString();
cd("ImportInitialValues.ResultGenerator"); getErrorString();
simulate(ImportInitialValues.ResultGenerator); getErrorString();

// Run ImportInitialValues.ImportModelAnnotationInitialEqs with relative path to init
cd(".."); getErrorString();
mkdir("ImportInitialValues.ImportModelAnnotationInitialEqs"); getErrorString();
cd("ImportInitialValues.ImportModelAnnotationInitialEqs"); getErrorString();
simulate(ImportInitialValues.ImportModelAnnotationInitialEqs, simflags="-iif=../ImportInitialValues.ResultGenerator/initial.mat"); getErrorString();

// Run executable from different folder
cd(".."); getErrorString();
system("./ImportInitialValues.ImportModelAnnotationInitialEqs/ImportInitialValues.ImportModelAnnotationInitialEqs -inputPath=ImportInitialValues.ImportModelAnnotationInitialEqs"); getErrorString();

Output

aheuermann1@DESKTOP-BQ04RIC:~/workspace/Testitesttest/ticket6035$ omc test.mos
0
""
true
""
true
""
"/home/aheuermann1/workspace/Testitesttest/ticket6035/ImportInitialValues.ResultGenerator"
""
record SimulationResult
    resultFile = "initial.mat",
    simulationOptions = "startTime = 0.0, stopTime = 1.0, numberOfIntervals = 500, tolerance = 1e-06, method = 'dassl', fileNamePrefix = 'ImportInitialValues.ResultGenerator', options = '', outputFormat = 'mat', variableFilter = '.*', cflags = '', simflags = '-r=\\'initial.mat\\''",
    messages = "LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
",
    timeFrontend = 0.004153400000000001,
    timeBackend = 0.0031308,
    timeSimCode = 0.0059271,
    timeTemplates = 0.0020405,
    timeCompile = 0.844584,
    timeSimulation = 0.0113651,
    timeTotal = 0.8713652
end SimulationResult;
""
"/home/aheuermann1/workspace/Testitesttest/ticket6035"
""
true
""
"/home/aheuermann1/workspace/Testitesttest/ticket6035/ImportInitialValues.ImportModelAnnotationInitialEqs"
""
record SimulationResult
    resultFile = "/home/aheuermann1/workspace/Testitesttest/ticket6035/ImportInitialValues.ImportModelAnnotationInitialEqs/ImportInitialValues.ImportModelAnnotationInitialEqs_res.mat",
    simulationOptions = "startTime = 0.0, stopTime = 1.0, numberOfIntervals = 500, tolerance = 1e-06, method = 'dassl', fileNamePrefix = 'ImportInitialValues.ImportModelAnnotationInitialEqs', options = '', outputFormat = 'mat', variableFilter = '.*', cflags = '', simflags = '-iif=../ImportInitialValues.ResultGenerator/initial.mat'",
    messages = "init_file flag value: ../ImportInitialValues.ResultGenerator/initial.mat
init_file: ../ImportInitialValues.ResultGenerator/initial.mat
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
",
    timeFrontend = 0.008947200000000001,
    timeBackend = 0.0159737,
    timeSimCode = 0.0025224,
    timeTemplates = 0.002746,
    timeCompile = 0.8144335,
    timeSimulation = 0.0190424,
    timeTotal = 0.8637486999999999
end SimulationResult;
""
"/home/aheuermann1/workspace/Testitesttest/ticket6035"
""
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
0
""

[Andreas Heuermann]

Never mind. Found the error in my test script and have solved the error.

system("rm -rf ImportInitialValues.ImportModelAnnotationInitialEqs* ImportInitialValues.ResultGenerator*"); getErrorString();

// Load model
loadFile("ImportInitialValues.mo"); getErrorString();

// Run ImportInitialValues.ResultGenerator
mkdir("ImportInitialValues.ResultGenerator"); getErrorString();
cd("ImportInitialValues.ResultGenerator"); getErrorString();
simulate(ImportInitialValues.ResultGenerator); getErrorString();

// Run ImportInitialValues.ImportModelAnnotationInitialEqs with relaive path to init
cd(".."); getErrorString();
mkdir("ImportInitialValues.ImportModelAnnotationInitialEqs"); getErrorString();
cd("ImportInitialValues.ImportModelAnnotationInitialEqs"); getErrorString();
simulate(ImportInitialValues.ImportModelAnnotationInitialEqs, simflags="-iif=../ImportInitialValues.ResultGenerator/initial.mat"); getErrorString();

// Run executable from different folder
cd(".."); getErrorString();
system("./ImportInitialValues.ImportModelAnnotationInitialEqs/ImportInitialValues.ImportModelAnnotationInitialEqs -inputPath=ImportInitialValues.ImportModelAnnotationInitialEqs -outputPath=ImportInitialValues.ImportModelAnnotationInitialEqs -iif=../ImportInitialValues.ResultGenerator/initial.mat"); getErrorString();

Added ​RP#6653.

[Adeel Asghar]

On a second thought I don't think it is a bug. You need to copy the file to %TEMP%/OpenModelica/OMEdit and not to %TEMP%/OpenModelica/OMEdit/ImportInitialValues.ImportModelAnnotationInitialEqs. The ​PR#6653 does fix the issue but is unnecessary.

I am adding Massimo Ceraolo to this conversation. He might remember the ticket where we decided to use OMEdit working directory for user defined input files.

[massimo ceraolo]

I think the whole thing was around #3788.

Maybe there was an additional ticket immediately later, but I cannot find any.

I remember we decided to use subfolders for individual simulations, while leaving the home directory (out of subfolders) for general usage files. What I needed were txt files, i.e. table inputs, and mentioned this in my comment 12 to #3788.

[Francesco Casella]

Replying to adeas31:

The ​PR#6653 does fix the issue but is unnecessary.

As I understand it, ​PR#6653 just makes sure -iif uses -inputPath to locate the init file. According to the ​documentation

-inputPath=value or -inputPath value

Value specifies a path for reading the input files i.e., model_init.xml and model_info.json

It seems pretty natural to me that what is called Equation System Initialization File in OMEdit Simulation Setup | Simulation flags is located in the same place as the .xml and .json files, which are also involving initialization.

The initialization of a model is a quite model-specific business, so I don't think it belongs to the base working directory. If we keep it in the model-specific directory, one can just call the file initial.mat, otherwise one would need to use long names as MyLibrary.MyModel.initial.mat to avoid ambiguities and overlapping in the common working directory.

I think the case of table input files is different. Normally, they should probably be placed in the library Resources directory and accessed through loadResource(), but for interactive use it makes sense to have them in the common temp directory, because the data file will probably have a name that refers to which data is in it, so there is no ambiguity. Also, many different models may share the same input data files, so it would be inconvenient to copy them in each and every model-specific directory. This is not the case for the initialization file, which is specific to a certain model.

@adeas31, you are right that when you run the simulation for the first time the model-specific directory is still not there. On the other hand, I see two typical use cases, and in both of them this won't be a problem.

Case 1.

You need to experiment different transients of a certain model, starting from the same initial condition, and changing some parameters that affect the transient. The initialization takes forever. So, you run it once (which creates the directory and the .mat file with the initial solution), then make a copy of the result file, call it init.mat and from that point onwards, you just load it to skip the 2-minutes initialization.

The problem in this case is that -iif currently overrides whatever values you can input in the Variables Browser, i.e. whatever values are provided in the _init.xml file. So, basically, unless you are some input sources depending on files that you can change independently, you are bound to run only run one transient, i.e. the one that was run in the first place. Which makes having a saved initial condition a bit pointless.

Do you think there could be some way to allow the end user to further override the imported values of some parameters through the GUI? I guess this would somehow require to save a modified .xml file as a result of the -iif process, so that you can then change it. This could be tricky to understand for an end-user that doesn't know how this feature is actually implemented in the runtime.

Case 2.

You have a complex model, created with another tool. You first try to simulate it, and if initializes successfully, there is no need to bother about initialization files. However, if it doesn't work, even after some failed attempts at fixing the initialization, then you may resort to the "doomsday option" of importing the initial values from the other tool simulation. Again, you will for sure do that after the directory has been created.

BTW, also in this case, not being able to change any parameters makes the feature a bit useless, because if you cannot change anything in the simulation, and you already have the results from another tool, what is the point of re-running a simulation in OpenModelica?

@ceraolo, what do you think?

[massimo ceraolo]

Regarding the position of table inputs in the OMEdit home directory, I think it was the right choice. For instance, I often simulate road vehicles, and in that folder, I keep test cycles, such as NEDC.txt and WLTC.txt (earlier and current test cycle used for homologation and tests).
Obviously, each of these files can be used along with different models and are therefore somehow model-independent. Having them in the home directory makes their usage very easy.

I've not used initial.mat yet with OM, but I agree that initial conditions should be linked to models, and therefore initial.mat should stay where the model results are. The situation here is indeed totally different from the case of table input text files.
Naturally, for the loading of initial.mat to be useful, there should be a mechanism allowing re-simulating with new parameters starting from a given saved initial.mat. I.e. one initial.mat, several simulations (each with its own non-structural parameters, set via GUI)

[Adeel Asghar]

Lets see it from the process flow point of view. Consider you don't have anything in the working directory and then you simulate ImportInitialValues.ResultGenerator, OMEdit creates a direcotry WorkingDirectory/ImportInitialValues.ResultGenerator and initial.mat is generated inside it. From here we have 2 alternatives,

1. Simply copy this file to working directory and simulate the model ImportInitialValues.ImportModelAnnotationInitialEqs without any error.
2. Simulate ImportInitialValues.ImportModelAnnotationInitialEqs so that OMEdit can create the folder WorkingDirectory/ImportInitialValues.ImportModelAnnotationInitialEqs. The simulation will fail because it can't find the initialization file. Now copy the file to WorkingDirectory/ImportInitialValues.ImportModelAnnotationInitialEqs and simulate again to get the proper result.

If alternative 2 is what you want then ​PR#6653 is correct.

[Francesco Casella]

@adeas31, the process flow of the MWE posted in this ticket is not of great interest, the MWE was only meant to reproduce the issue without involving tons of code.

If you consider real-life use-cases, as mentioned in comment:7 and comment:8, the benefits of alternative 2 outweigh the shortcomings.

Hence, I just merged in the PR and we can close this ticket.

The real issue, however, is that those real-life use cases cannot be supported even with ​PR 6653, because once you import results with -iif, you cannot change the parameter values from the GUI, as these changes affect the _init.xml file which is ignored by the runtime because of -iif.

I just opened #6039 on this topic, I guess we'll need to handle that for the next release.