Modify

Opened 6 years ago

Last modified 6 years ago

#1242 new defect

test system does not pause if test fails to run

Reported by: Roland Haas Owned by:
Priority: minor Milestone:
Component: Cactus Version:
Keywords: Cc:

Description

when running all tests in a thorn, the test system does not stop for user input when a test completely fails to run (or produces no output files) eg due to an error in the parameter file.

  Test GRHydro: tov_carpetevolutionmask2
    "Carpet mesh refined, one TOV, Gamma-law, use evolution mask"

  Issuing mpirun -np 2 exe/cactus_bns_all arrangements/Zelmani/GRHydro/test/tov_carpetevolutionmask2.par
Cactus exited with error code 1
Please check the logfile TEST/bns_all/GRHydro/tov_carpetevolutionmask2.log

  
  No files created in test directory

  Failure: 6 files missing, 0 files compared, 0 differ

Had the test run but produced different files, the system would have stopped allowing he user to inspect the results.

For a thorn with many tests this can make it hard to spot these errors.

Attachments (0)

Change History (3)

comment:1 Changed 6 years ago by Frank Löffler

Instead of stopping I would probably prefer here to produce a summary of the test results in the end, like what is done when running all tests. What do you think?

comment:2 Changed 6 years ago by Roland Haas

Having it produce a summary would be even more helpful. Having to stop at each failing test when running all in a thorn can be occasionally tedious, yes.

comment:3 Changed 6 years ago by Ian Hinder

I always found it very strange that the system behaved differently if you asked it to run all tests or not. I think the summary should be output in all cases. That being said, when SimFactory runs the tests, it always runs "all" tests. The reason is that it is always noninteractive, and the summary log is useful. Additionally, because SimFactory needs to have all the input data available at the time the job runs, i.e. it must not rely on having the home filesystem available and in the same state as when the job was submitted, it actually copies the reference data for the requested tests into the job output directory. This is how it filters which tests are supposed to be run, and then it can run "all tests" and generate the summary output. I always use simfactory when running tests, as I don't find the interactive prompting very useful, preferring to examine the output and diffs etc myself in a text editor.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
Next status will be 'confirmed'.
The owner will be changed from (none) to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.