Checkpoint recovery nonfunctional

Comments (14)

1. 

   Ian Hinder reporter

   • removed comment

   Workaround: recovery can be made to work if you add the options

   --from-restart-id=<n> --recover

   to the submit command, where <n> is the number of the restart that you want to recover from.

   • 2011-02-27T11:10:34+00:00
2. 

   anonymous

   • removed comment

   Ian, I have committed a fix for this. Please test without using --from-restart-id or --recover.

   • 2011-03-11T13:06:51+00:00
3. 

   Ian Hinder reporter

   • removed comment

   I cannot get this to work on my laptop. It still doesn't seem to be linking the checkpoint files into the new restart directory. Do you think it depends on the machine that I run on? I have tested this on my laptop. What machine have you tested this on, so that I can try on the same machine?

   • 2011-03-11T17:06:27+00:00
4. 

   anonymous

   • removed comment

   I have attached a patch to fix checkpointing. please test.

   • 2011-03-14T12:03:06+00:00
5. 

   Erik Schnetter

   • removed comment

   The patch looks good; since Ian confirmed that it solves the problem, please apply it.

   • 2011-03-14T15:00:07+00:00
6. 

   Ian Hinder reporter

   • removed comment

   It looks like it only attempts to recover from the previous restart. What happens if that restart never ran, perhaps because it was deleted from the queue, and hence there are no checkpoint files in it? I think simfactory should try to recover from the last restart which actually has checkpoint files, not just the last restart. If there are no restarts with checkpoint files, then it should not try to recover.

   I don't think the patch should be applied until it works for the case where a job never ran, and hence there are no checkpoint files in the last few restarts.

   Aside: all of this would have been trivial if we used a single common directory for checkpoint files, common across restarts. All the logic is present in Cactus already.

   • 2011-03-15T08:09:59+00:00
7. 

   Erik Schnetter

   • removed comment

   Let's discuss this on the mailing list. Using a common checkpoint directory may be a good idea.

   • 2011-03-15T08:42:52+00:00
8. 

   anonymous

   • removed comment

   I have attached a new patch that finds all checkpointing files and links them. Please review.

   • 2011-03-18T13:27:13+00:00
9. 

   Erik Schnetter

   • removed comment

   This patch has several problems:

   We just agreed on the mailing list to store checkpoint files in a directory "checkpoints" that is at the same level as the "output-NNNN" directories. These checkpoint files therefore do not fit the pattern you are expecting, and lead to warnings. Instead, these checkpoint files should be ignored. Instead of looking for checkpoint files in the whole simulation directory, it may be better to look in the individual restart directories.

   The code to replace the "output-NNNN" patterns is too complex. Instead of pattern matching and then manual string operations, a direct regexp replacement would be simpler and safer.

   The pattern matching code does not check where in the path name the pattern "output-NNNN" exists. If a simulation is called "output", this leads to problems.

   The message "linking file XXX" is printed even if the linking step does not actually happen.

   The code to check whether a checkpoint file is found twice only looks at file names. This is fragile and can hide problems, for example when two restarts use different numbers of processes. Instead, the incoming list of checkpoint files should be pruned. This would also allow an explicit choice of whether newer or older checkpoint files should be chosen if they exist in several restarts: the current code presumably chooses the older ones, whereas we may want to use the newer ones.

   The fallback call to "shutil.copyfile" overwrites existing file contents. This is a serious problem if the existing file is a hard link (because this will then overwrite the file content). The destination needs to be unlinked first.

   This code links all checkpoint files. Only checkpoint files from the last iteration should be linked.

   Overall, the code says "checkpoint" when it should say "recover" in several places.

   • 2011-03-18T14:01:15+00:00
10. 

    Frank Löffler

    • changed milestone to ET_2011_05
    • removed comment

    • 2011-03-27T23:09:02+00:00
11. 

    Frank Löffler

    • changed milestone to ET_2011_11
    • removed comment

    • 2011-03-28T11:02:20+00:00
12. 

    Erik Schnetter

    • removed comment

    Note: I have just successfully submitted three restarts of a simulation on Damiana, using a global "checkpoints" directory.

    • 2011-05-23T17:18:54+00:00
13. 

    Erik Schnetter

    • changed status to resolved
    • removed comment

    Note: In my comment above, I meant "finished" instead of "submitted".

    I have also just successfully finished three restarts of a simulation on Damiana, without using a global "checkpoints" directory.

    Since the problem seems to not exist any more, I am closing this ticket.

    • 2011-05-23T18:21:54+00:00
14. 

    Roland Haas

    • edited description
    • changed status to closed

    • 2019-02-21T20:25:02+00:00
15. Log in to comment