Opened 7 years ago

Last modified 4 years ago

#727 assigned defect

Test output changes every time some tests are run

Reported by: Ian Hinder Owned by:
Priority: major Milestone:
Component: EinsteinToolkit thorn Version:
Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH Cc:


Each time a new run of the test suites is performed, the test output of IDAxiOddBrillBH and QuasiLocalMeasures changes a little. These changes are below the tolerances set in the test.ccl files, so the tests pass, but there should be no change at all when the tests are run on the same machine.

An example of the difference from one test run to the next is shown at

Another example, from QuasiLocalMeasures, is at

The only thorns to exhibit this behaviour are IDAxiOddBrillBH and QuasiLocalMeasures. All other test data remains unchanged from one test run to the next. Note that the two test runs will be performed with different executables, since a new configuration is created for each run. I think I remember testing that simply re-running the same executable resulted in the same test output.

Is it possible that there are a small number of points which access uninitialised memory which changes on each run for these thorns?

Attachments (0)

Change History (6)

comment:1 Changed 7 years ago by Erik Schnetter

Since the output changes only when the executable changes, it may be that the compiler is generating different code. Certain optimisations may depend on the order in which certain internal compiler data structures are laid out, which may depend e.g. on the time and date.

Can you save two executables? I would like to compare the disassembled output of the routine generating these data, and see whether they differ.

comment:2 Changed 7 years ago by Ian Hinder

I have removed the "rm" command, so executables will be saved from now on in the automated system. I will make them available when they appear.

comment:3 Changed 7 years ago by Erik Schnetter

Ian, can you provide this additional feedback now?

comment:4 Changed 7 years ago by Ian Hinder

Owner: set to Ian Hinder
Status: newaccepted

comment:5 Changed 7 years ago by Ian Hinder

I have seen something similar to this recently as well. The test GRHydro_test_tov_ppm_ML actually passed on one run and failed when it was run again. The only commit between these two modified unrelated test data and parameter files, and couldn't possibly have changed the behaviour of GRHydro.

The first run was and the test passed ( The second run was and the tests failed (

The only commit between these two runs was which could not change the results for the GRHydro test.

Note that this test has been consistently passing when run using the old test system (

The executables are available in datura:/lustre/datura/ianhin/projects/temp.

comment:6 Changed 4 years ago by Ian Hinder

Owner: Ian Hinder deleted
Status: acceptedassigned

Modify Ticket

Change Properties
Set your email in Preferences
as assigned The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
The owner will be changed from (none) to anonymous.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.