Opened 5 years ago

Last modified 5 years ago

#1312 new enhancement

quick rsync for simfactory

Reported by: Frank Löffler Owned by: Erik Schnetter
Priority: major Milestone:
Component: SimFactory Version: development version
Keywords: Cc:


Quite often I find myself developing something (code/parfiles) locally and use 'sim sync' to get it to a remote location. Until a recent change in simfactory this was usually quite fast (a couple of seconds at most). In all of these cases I know that I didn't change anything on the remote side. Now 'sim sync' can take a minute even to local machines, depending on their file system load since the remote side has to actually read the complete tree for comparison. And this is without any actual changes.

I propose to have a 'sim sync --fast' option which uses timestamps as before. This should not be the default since it might not catch all changes in every situation, but it would make quick development sessions like this (edit locally, compile and run remotely) feasible again.

Attachments (0)

Change History (4)

comment:1 Changed 5 years ago by Erik Schnetter

Simfactory already has "sync --parfiles", which copies only parfiles and is thus much faster.

What about the following idea: Simfactory first uses timestamps to copy files, optimistically assumes that this led to the correct results and begins the build on the remote system, but continues with a full (slower) sync as well. If the full sync exposes a problem, the build is aborted. Otherwise, the build result is accepted.

What is your target system? Would it make sense to set up e.g. DropBox or Google Drive to keep the source tree in sync? These are faster since they begin to copy while you edit, and monitor (via the operating system) which files are being changed.

comment:2 Changed 5 years ago by Frank Löffler

Parfiles was just one example. It might be a regular source file as well.

I don't really like the idea of the test for a successful build. A build might be successful even when a file is not updated (yet) and thus the exe contains the wrong code in the end (built earlier than the parallel sync was copying the new file).

In this case the system is supermike at LSU, but that is even one of the faster machines. I don't like setting up a system like DropBox or Google drive on a cluster head node. Not only do I worry about admins disliking this, I also don't like to put my passwords on these machines.

comment:3 Changed 5 years ago by Frank Löffler

Also, I tend to use the local simfactory just for the rsync and then the remote simfactory (within a second terminal, logged in remotely) for the build.

comment:4 Changed 5 years ago by Ian Hinder

As someone who never edits on the remote system, I would also benefit from such a "quick" option. I would like to be able to set it in my configuration and obtain the old behaviour. I also dislike the proposal for detecting a failed build and syncing in the background. It is too complicated and error-prone, in my opinion.

Modify Ticket

Change Properties
Set your email in Preferences
as new The owner will remain Erik Schnetter.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from Erik Schnetter to the specified user.
Next status will be 'confirmed'.
The owner will be changed from Erik Schnetter to anonymous.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.