quick rsync for simfactory

Comments (5)

1. 

   Erik Schnetter

   • removed comment

   Simfactory already has "sync --parfiles", which copies only parfiles and is thus much faster.

   What about the following idea: Simfactory first uses timestamps to copy files, optimistically assumes that this led to the correct results and begins the build on the remote system, but continues with a full (slower) sync as well. If the full sync exposes a problem, the build is aborted. Otherwise, the build result is accepted.

   What is your target system? Would it make sense to set up e.g. DropBox or Google Drive to keep the source tree in sync? These are faster since they begin to copy while you edit, and monitor (via the operating system) which files are being changed.

   • 2013-04-09T11:00:31+00:00
2. 

   Frank Löffler reporter

   • removed comment

   Parfiles was just one example. It might be a regular source file as well.

   I don't really like the idea of the test for a successful build. A build might be successful even when a file is not updated (yet) and thus the exe contains the wrong code in the end (built earlier than the parallel sync was copying the new file).

   In this case the system is supermike at LSU, but that is even one of the faster machines. I don't like setting up a system like DropBox or Google drive on a cluster head node. Not only do I worry about admins disliking this, I also don't like to put my passwords on these machines.

   • 2013-04-09T11:21:37+00:00
3. 

   Frank Löffler reporter

   • removed comment

   Also, I tend to use the local simfactory just for the rsync and then the remote simfactory (within a second terminal, logged in remotely) for the build.

   • 2013-04-09T11:23:23+00:00
4. 

   Ian Hinder

   • removed comment

   As someone who never edits on the remote system, I would also benefit from such a "quick" option. I would like to be able to set it in my configuration and obtain the old behaviour. I also dislike the proposal for detecting a failed build and syncing in the background. It is too complicated and error-prone, in my opinion.

   • 2013-04-09T15:25:39+00:00
5. 

   Roland Haas

   Such a quick option is very dangerous. Editing on the remote system is not required to end up with an inconsistent state.

   It can be triggered by the following steps:

   1. use sim sync to copy to a remote cluster
   2. edit on your local machine
   3. use sim build on the remote cluster
   4. use sim sync to copy to the remote cluster once more

   Since 3 happened after 2 the object files created on the cluster now have a timestamp newer than the source files on the local machine. Now since the --times option (needed to make timestamp based skipping possible) also causes rsync to set the time stamp of the copied files to the timestamp on the source system in 4 (see https://linux.die.net/man/1/rsync), make will not consider the object file to be out of date wrt to its source file (that was just copied in).

   So a sim build on the cluster after 4 will not recompile files even though those files were changed (and rsync in fact copied them because the time stamp differed).

   You can currently get back the behaviour you desire by setting your rsyncopts to --times in your defs.local.ini. If adding such a qick option I would try and name it --unsafe to make very clear to anyone using it that it hands out rope and you may now hang yourself.

   Note that steps 1 -- 4 can very easily happen if you have multiple remote machines if you every compile without doing a sim sync just before.

   • 2020-04-03T21:16:37+00:00
6. Log in to comment