Push testsuite results to a git repository

Comments (4)

1. 

   Ian Hinder

   • removed comment

   I have been doing this in the development autotest system. The system runs the tests on both 1 and 2 processes in separate simfactory simulations (in the queuing system), waits for them to finish, and then commits the output directories (i.e. TEST/<config>) to a results repository. The results repository is hosted on git.barrywardell.net, and there is a gitweb interface to it. You can see this at http://git.barrywardell.net/EinsteinToolkitTestResults.git. Click on "tree" to see the results of each run. This contains the build log, the log from the autotest system, and the TEST directory from the 1 and 2 process runs (including the summary log and all output). In future, I would do this on multiple machines, and the results from each machine might be pushed to separate branches in the same repository.

   Comparing your suggestions with my approach, there are a couple of differences.

   1. I identify a set of simulations run on different numbers of processes as a "test run", as this is the logical unit that is always tested together. Having to manage sets of simulations is harder than managing a single "test run" composed of multiple simulations.

   2. I find it useful to have a single repository where each commit represents a different test run. This allows you to track any changes between runs using standard git tools. If we also had separate branches for different machines, then the usual git tools would also allow you to compare between branches. This also means that you cannot expire old runs from the repository, and the repository grows in size without limit. Git only stores one copy of identical files, and also does delta compression between revisions. Both of these should significantly mitigate the size requirement. For reference, with 214 test runs, the repository is 363 MB.

   3. I'm not sure if you were intending to store just the summary.log files or also the test output data in the repository. Maybe it would be enough to store the log files and the diffs, rather than the output data itself, though having the output data has been useful to me in the past

   If this was implemented in Cactus, I would not have to wait for the simulations to finish, as once they were submitted, they would be autonomous. This would be a good thing. In order to have a single commit for both 1 and 2 process tests, each simulation could check whether it was the last one to run, and only the last one would do the commit.

   git might not be available on the compute nodes where the test suites are run. This is true on Datura. In principle we could build git as part of the ET and store the executable in the simulation directory for systems where this was the case.

   • 2012-05-16T02:28:44+00:00
2. 

   Erik Schnetter reporter

   • removed comment

   Would it make sense to have each test run as a single simulation with multiple restarts? This would fit better into the Simfactory scheme, but would prevent running them in parallel.

   I usually have branches for configurations (not machines). That means that e.g. damiana-gcc and damiana-intel would be in different branches.

   It is possible to expire old runs. This would probably make it necessary to not link the different commits into a sequence (but rather have a multi-headed repository). Each commit would also be tagged. Removing the tag would then expire a run. This may not be a good idea, because it may confuse many tools. I don't do this in Formaline.

   I just checked, and a single test result uses 8 MB. This indicates an incremental cost of about 2 MB per test run. I assume you run "git gc" from time to time, and that 363 MB is measured after a gc?

   I would store the output data as well. If we run out of space, we can reduce the size (but moving some of the files to a new repository).

   • 2012-05-16T11:59:31+00:00
3. 

   Ian Hinder

   • removed comment

   Would it make sense to have each test run as a single simulation with multiple restarts? This would fit better into the Simfactory scheme, but would prevent running them in parallel.

   At the moment, compilation on Datura takes about 10 minutes (on a good day), and each test run also takes about 10 minutes. So yes, it would be useful to run them in parallel (30 mins down to 20 mins), but not essential. Performance on other machines will vary, but I expect the compilation time to dominate (we have a relatively fast home filesystem). This is, of course, an abuse of the simfactory restart mechanism. We could alternatively teach simfactory to run multiple test sets in a single simulation. It already has special code for dealing with test suites. This would need some tweaking of mpirun/nprocs settings.

   I usually have branches for configurations (not machines). That means that e.g. damiana-gcc and damiana-intel would be in different branches.

   It is possible to expire old runs. This would probably make it necessary to not link the different commits into a sequence (but rather have a multi-headed repository). Each commit would also be tagged. Removing the tag would then expire a run. This may not be a good idea, because it may confuse many tools. I don't do this in Formaline.

   Having separate heads would indeed enable run expiry. Why would tools get confused? What we could also do is start a new head every time the tests pass successfully. We are not usually interested in history before that point any more. We could then expire old heads periodically if we wanted to.

   I just checked, and a single test result uses 8 MB. This indicates an incremental cost of about 2 MB per test run. I assume you run "git gc" from time to time, and that 363 MB is measured after a gc?

   Yes, I ran a repack and a gc before measuring the repo. Does your 8 MB include both 1 proc and 2 proc results? That is quite a large incremental cost. It should be much smaller than that. It is probably not an incremental cost, but the cost of regenerating test data from time to time. There are also one or two very fishy tests which generate different numerical results each time they are run (but within tolerance). Having test results as separate commits in the same branch allowed me to detect this sort of behaviour.

   I would store the output data as well. If we run out of space, we can reduce the size (but moving some of the files to a new repository).

   Or just starting a new repository in a state where all the tests pass, which could correspond to a 6-monthly release.

   • 2012-05-16T16:45:50+00:00
4. 

   Barry Wardell

   • removed comment

   Replying to [comment:3 hinder]:

   >Would it make sense to have each test run as a single simulation with multiple restarts? This would fit better into the Simfactory scheme, but would prevent running them in parallel.

   At the moment, compilation on Datura takes about 10 minutes (on a good day), and each test run also takes about 10 minutes. So yes, it would be useful to run them in parallel (30 mins down to 20 mins), but not essential. Performance on other machines will vary, but I expect the compilation time to dominate (we have a relatively fast home filesystem). This is, of course, an abuse of the simfactory restart mechanism. We could alternatively teach simfactory to run multiple test sets in a single simulation. It already has special code for dealing with test suites. This would need some tweaking of mpirun/nprocs settings.

   Have you tried using ccache to speed up the compile process? It can make a surprisingly big difference to compile times in some cases. It might be that we are so filesystem limited that it's not worth it, but it's worth a try and much easier than trying to implement parallel tests.

   Replying to [comment:3 hinder]:

   Having separate heads would indeed enable run expiry. Why would tools get confused? What we could also do is start a new head every time the tests pass successfully. We are not usually interested in history before that point any more. We could then expire old heads periodically if we wanted to.

   Or just starting a new repository in a state where all the tests pass, which could correspond to a 6-monthly release.

   There isn't really any difference between having separate heads in the same repository and having separate repositories other than the convenience of being able to diff between revisions in a single repository. It's also straightforward to go from the separate heads case to the separate repositories case and back again.

   Replying to [comment:3 hinder]:

   Having test results as separate commits in the same branch allowed me to detect this sort of behaviour.

   You could still detect this without the commits having to be on the same branch, couldn't you?

   • 2012-05-16T17:04:09+00:00
5. Log in to comment