Fold option list, submit script, and run script into mdb

Issue #761 new
Erik Schnetter created an issue

It would be nice to avoid having both an mdb, and three separate files (option list, submit script, run script) for each machine. These really belong together.

The following may be possible: 1. An option list consists of a set of key-value pairs. It should be straightforward to put these into the mdb. 2. The submit script consists essentially of a single command, and various options that are set via comments. These options could also be set via command line options to qsub, and thus be stored in the mdb. In fact, Orca (Sharcnet) already requires this, and only has a trivial submit script. 3. All run scripts follow the same pattern: some commands to change the environment, some commands to mangle MPI node lists, and an mpirun command. Interspersed is a lot of boilerplate. These could be abstracted out into two or three MDB entries specifying these commands.

The result would be that all information is stored in the mdb, and no additional files need to be tracked.

To protect against mdb modifications while a job is waiting, we could/should store a copy of the mdb entry with each simulation.

If the current mdb format is not convenient for storing this information (but why should it not?), then we could examine other options, e.g. based on XML or SqLite. (There are e.g. XML-like formats that have the same semantics as XML, but use a slightly different syntax that is friendlier for humans, but is as easy to parse for machines.) XML and friends would give us a hierarchy, which INI doesn't.

Keyword:

Comments (1)

  1. Roland Haas
    • removed comment

    I agree that the SubmitScript/RunScript separation is artificial (and unusual). I am not sure though if I would really like to have them folded into a large set of options inside of the machine database. The current setup has (I believe) that advantage that the SubmitScript/RunScript is recognizably a qsub script for which one can usually find examples and documentation on new clusters. Having to learn a whole new syntax of option to encode this information into simfactory would seem quite a bit of extra complication unless the benefits are large. With Cactus' option files this might be less of a problem. The only change would be that the boundary between simfactory and the low-level Cactus build system blurs (which might actually be what one wants).

  2. Log in to comment