Opened 7 years ago
Closed 7 years ago
#4880 closed defect (fixed)
Misleading regression analysis
| Reported by: | Francesco Casella | Owned by: | Martin Sjölund |
|---|---|---|---|
| Priority: | high | Milestone: | 1.13.0 |
| Component: | Testing Framework | Version: | |
| Keywords: | Cc: |
Description
The new scripts are often misleading when the indicate regressions or improvements. For example, take this one. The two "improvements" are obviously bogus. The fact that the front end is broken and reports 0 execution time is a regression, not an improvement.
On the other hand, if any of the phases (e.g., frontend) aborts before the end, an increase of the time may actually mean an improvement, because it may mean that the phase completes more steps and fails later.
I would suggest to show performance regressions and improvements only for those phases that are successfully completed before and after the commit.
Change History (1)
comment:1 by , 7 years ago
| Milestone: | Future → 1.13.0 |
|---|---|
| Resolution: | → fixed |
| Status: | new → closed |
Note:
See TracTickets
for help on using tickets.
Done (but only if it aborts before code generation finishes; this should avoid these annoying emails)