Modify

Opened 7 years ago

#522 new enhancement

Improve consistency and performance of sync/build/create

Reported by: Ian Hinder Owned by: Erik Schnetter
Priority: minor Milestone:
Component: SimFactory Version:
Keywords: Cc:

Description

One of the design goals of SimFactory is to make it very difficult for a user to make mistakes or end up with inconsistent results. Currently, if a user modifies a source file and submits a simulation without remembering to rebuild the configuration, the old version of the configuration will be used. SimFactory could scan all the source files which would be used for a build and warn the user if there are new source files. Optionally it could also rebuild the configuration automatically. This check might be better implemented in Cactus, which in principle has the dependency information already. It is functionally equivalent to performing a build before every create, so maybe the best approach would be to make the Cactus build process much faster in the case where no files have been modified. This may not be possible.

A related idea is to ensure that remote configurations are synced before being built, and parameter files are synced before simulations are created. One could record the time of the last successful sync (separately of the source tree and the parameter files) to each machine, and look for files which have been modified since that time. If there are such files, a sync could be initiated, or the user warned. Statting files on a laptop should be very fast. Statting files on Kraken can take minutes or hours. This information could also be used to restrict the list of files synced, to avoid having to stat every file in the source tree on the remote system. This assumes that the only modifications of the source tree on the remote system are due to simfactory on the local system. For me, this is the case 99.999% of the time.

I attach a script that I wrote to interact conveniently with remote systems. It performs a sync of either the source tree or the parameter files depending on the simfactory command you want to run. It allows you to use the syntax

<machine> <command> <args>

instead of

sim --remote <machine> <command> <args>

e.g.

damiana list-simulations

The script should be somewhere on your path, and symlinks named after machines should be created pointing to it. For example,

cp machine .../bin
chmod u+x .../bin/sim/machine
cd .../bin
ln -s machine datura
ln -s machine damiana
ln -s machine kraken

cd ~/Cactus

datura sync
datura create-submit ...
datura build

Attachments (1)

machine (544 bytes) - added by Ian Hinder 7 years ago.

Download all attachments as: .zip

Change History (1)

Changed 7 years ago by Ian Hinder

Attachment: machine added

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain Erik Schnetter.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from Erik Schnetter to the specified user.
Next status will be 'confirmed'.
The owner will be changed from Erik Schnetter to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.