- removed comment
Tests should use IO::out_fileinfo = "none"
Issue #702
closed
The Cactus User Guide (http://einsteintoolkit.org/documentation/UsersGuide/UsersGuidech9.html#x13-117000C1.8) recommends that test output files should always be the same, and hence use IO::out_fileinfo = "none". I would like to implement this for the tests in the ET, as it makes comparing test output using standard (non-Cactus testsuite mechanism) diff tools possible.
Keyword:
Comments (11)
-
-
- changed status to open
- removed comment
-
- changed status to open
- removed comment
-
reporter
- removed comment
Replying to [comment:1 eschnett]:
Please go ahead. Ideally, you would generate the new test case output by removing lines from the old output, but that may be too much to ask for.
Since the changes should be invisible to the Cactus test mechanism, it should be enough to rerun the tests with the new output, verify that they still pass, then commit the new data. We can then use version-control tools to examine the diff, which should consist only of these header lines.
-
- removed comment
Has this been taken care of already?
-
- removed comment
Replying to [comment:4 hinder]:
Replying to [comment:1 eschnett]:
Please go ahead. Ideally, you would generate the new test case output by removing lines from the old output, but that may be too much to ask for.
Since the changes should be invisible to the Cactus test mechanism, it should be enough to rerun the tests with the new output, verify that they still pass, then commit the new data. We can then use version-control tools to examine the diff, which should consist only of these header lines.
I would actually be very surprised if the differences were only the header lines. Some of those test files were generated years ago with a compiler and options that are no longer in use. I would very much expect that regenerating the data would change the data in the files (this already happens when comparing intel and gcc compiler results).
-
- changed status to resolved
- removed comment
Applied in git hash:
b38107f392a1 of CactusExamples ec2db396097d of CactusNumerical de1f30b60874 of EinsteinEvolve 8e3226456e90 of EinsteinInitialData 7481f264b995 of PITTNullCode
-
- removed comment
Replying to [comment:6 rhaas]:
Replying to [comment:4 hinder]:
Replying to [comment:1 eschnett]:
Please go ahead. Ideally, you would generate the new test case output by removing lines from the old output, but that may be too much to ask for.
Since the changes should be invisible to the Cactus test mechanism, it should be enough to rerun the tests with the new output, verify that they still pass, then commit the new data. We can then use version-control tools to examine the diff, which should consist only of these header lines.
I would actually be very surprised if the differences were only the header lines. Some of those test files were generated years ago with a compiler and options that are no longer in use. I would very much expect that regenerating the data would change the data in the files (this already happens when comparing intel and gcc compiler results).
Are we here talking about tests that haven't run in a while? All the ones that are regularly run should still pass after this change, shouldn't they? They should be tested, using current compilers, all the time, and except the ioinfo, none of the generated data should change by this source change alone.
-
reporter
- removed comment
The newly-generated output may be different to that which is currently stored as reference data. It should agree within the tolerances, but it will likely look different.
-
- removed comment
Yes, that is true. It even likely would look different on different machines, or using different compilers or compiler options. And this could even lead to new test suite failures, on some machines.
-
- edited description
- changed status to closed
- Log in to comment
Please go ahead. Ideally, you would generate the new test case output by removing lines from the old output, but that may be too much to ask for.