sim setup should create a section for the current machine

Issue #1946 open
Roland Haas created an issue

When calling sim setup on a machine in the machine data base (eg bluewaters in my case) it it still creates only a [default] section in defs.local.ini which means that the account information and possibly non-default sourcebasedir are not used for bluewaters leading to hard to understand error messages later on.

I would thus like to suggest to change sim setup such that instead (or in addition to) a "[default]" section it creates a section for the detected machine if it already knows this machine.

Keyword:

Comments (10)

  1. Ian Hinder
    • removed comment

    Unfortunately, modifying "sim setup" is very awkward. Instead of direct python code which asks questions and sets up a data structure which is eventually written to defs.local.ini, which would be easy to modify, the questions and actions are defined in terms of a "decision tree" via configuration files. This means that adding logic which was not originally intended to the setup process requires the additional step of adding such logic to the generic decision tree language, then implementing the change in the configuration file. "sim setup" starts off in sim-util.py in command_setup, where we have

    dt = simdt.DecisionTree("defaults")
    dt.setupTree()
    dt.begin()
    

    Here, "defaults" refers to the file lib/dt/defaults.ini, which contains the configuration for the "defaults" decision tree. It contains things like

    [action:1]
    prompt = Determining local machine name
    action = %GET_LOCAL_MACHINE_NAME()%
    dest = @machine@
    printresults = yes
    

    which means that the code should ask for the local machine name, then execute a "macro" called GET_LOCAL_MACHINE_NAME (maybe for the default?), then store the result in a section of the ini file, expanded as @..@. I'm not sure which machine is referred to here.

    Anyway, there is a facility for setting machine-specific options; this is the question at the end of the setup process which asks if you want to set explicit machine-specific options. One way to work around the current problem would be to ask users to set up their sourcebasedir here, but this is obviously suboptimal.

    Perhaps what we want is to copy all the settings from [default] to [<machine>] after the setup process completes (though presumably not overriding any which had been explicitly set by the user). If we could access the ini file from command_setup, we might be able to try this, but by the time dt.begin() exits, the file has already been written. There's also no way to hook in to the system between writing the [defaults] and [<machine>] sections.

    In the spirit of the original code, we probably want a concept of copying everything that was set in one part of the decision tree ([default]) into another part ([<machine>]).

    One suggestion would be to replace the entire "sim setup" code with a simpler implementation. Unless I am missing something, it seems like it should be less than a couple of pages of code. We might lose some "advanced" functionality, but I suspect we don't use that functionality anyway.

    Is anyone already familiar with this code, and would like to take a stab at fixing the problem in a way which is consistent with the original implementation? Erik?

  2. Roland Haas reporter
    • removed comment

    Now that the release is done, would someone want to review this (Ian, Erik are candidates for reviewers).

  3. Roland Haas reporter
    • changed status to open
    • removed comment

    Making a [<machine>] section is actually dangerous on the clusters. Simfactory's setup command (both the interactive and the silent variant) create settings for sourcebasedir (probably ok since they set it to the parent of the current directory) and also basedir, and the latter one is likely wrong since the default that simfactory uses is $HOME/simulations which is almost certainly wrong on clusters.

    So basically we have the situation where half of the information is what we want and half is not.

    The fix is likely to try and see if one can make the defaults be the values from an existing machine.ini file if one find oneself on a known machine. The tricky part is probably to try and see if one can get a mdb table for the current machine without an existing defs.local.ini.

  4. Steven R. Brandt
    • removed comment

    Roland, what should we do with this ticket? What are you currently doing on BW?

  5. Roland Haas reporter
    • removed comment

    I think the code proposed (if there is any) should be changed to either pre-populate from the existing machine entry or only set "safe" values in the machine entry (eg user name, allocation, maybe sourcebasedir).

    Right now on BW I manually edit defs.local.ini after simfactory's setup code is done.

  6. Roland Haas reporter

    @Steven R. Brandt also came across this issue of sourcebasedir not being set when running sim setup-silent on known machines and in a directory different from what simfactory has in its machine database (private communication).

    This ticket may serve as a catchall to collect discussion and possible patches.

    The issue does not occur in the “official” workflow (see SF paper: https://arxiv.org/pdf/1008.4571 page 4, “A. Managing Source Code“), which would avoid ever running sim setup on the cluster (and never run GetComponents there either).

  7. Roland Haas reporter

    It’s a failed attempt at a fix. I may be adaptable though I am not really sure. The problem is that, as written, it also sets basedir which should not be done. The pull request would have duplicated the created default section as a section for the current machine, which is too much. Basically there are only some options that one wants to overwrite (one must overwrite since the options exist in the machine entry, obvious for basedir but also, annoyingly, for things like allocation which is set to NO_ALLOCATION for most machines that do require an allocation).

  8. Log in to comment