id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
4880	Misleading regression analysis	Francesco Casella	Martin Sjölund	"The new scripts are often misleading when the indicate regressions or improvements. For example, take [https://libraries.openmodelica.org/branches/history/newInst/2018-04-11%2011:16:42..2018-04-11%2015:07:03.html this one]. The two ""improvements"" are obviously bogus. The fact that the front end is broken and reports 0 execution time is a regression, not an improvement.

On the other hand, if any of the phases (e.g., frontend) aborts before the end, an increase of the time may actually mean an improvement, because it may mean that the phase completes more steps and fails later.

I would suggest to show performance regressions and improvements only for those phases that are successfully completed before and after the commit."	defect	closed	high	1.13.0	Testing Framework		fixed