1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

SimFactory home directory on Kraken is too specific

Issue #382 closed
Ian Hinder created an issue

The mdb entry for Kraken in SimFactory has

'sourcebasedir' => '/nics/b/home/@USER@',

My home directory is

'/nics/d/home/@USER@'

Either we could leave the source base dir as unset to force the user to set it, or we could automatically detect the location of the user's home directory (better).

Keyword:

Comments (20)

  2. Frank Löffler
    • changed status to open
    • removed comment

    First of all, sourcebasedir of kraken is not set to the home-dir in simfactory anymore, at least not in the python version.

    Secondly, you should be able to set it to \$HOME, assuming that $HOME is defined on the remote machine, which it typically is. We could do the same for SCRATCH and WORK, and possibly PROJECT which some machine environments by default provide.

    Should we leave the ticket open to change machines which could use this? Can someone else than me try this mechanism? (works for me)

  3. Erik Schnetter
    • removed comment

    I don't think that setting it to \$HOME will work. Simfactory will look at this directory in Python code, and there is nothing that expands \$HOME to the corresponding environment variable. There is a syntax for accessing environment variables maybe @ENV[HOME]@; you could try this. We then also need to check that Simfactory expands this wherever it is used.

    The current Kraken setting is "/nics/a/proj/cactus/@USER@/xt5", which is probably not a good default, as it is specific to the LSU relativity group and our collaborators. If someone confirms that using the home directory works for checking out and building the default Einstein Toolkit thornlist, then let's change this.

  4. Ian Hinder reporter
    • removed comment

    As far as I know, the syntax for expanding environment variables is not implemented. You can see in the code where it was going to go, but it was never actually added.

  5. Frank Löffler
    • removed comment

    \$HOME at least works for syncs, where this is directly given to the command line (with the backslash removed), so it's expanded on the shell of the remote machine. Now of course, if that directory variable should also be used inside python that would be a problem.

  6. Roland Haas
    • removed comment

    the same issue occurs with the $WORK directories on ranger and lonestar. Mine is /work/00945/@USER@ while simfactory uses /work/00507/@USER@. Using \$HOME/work or \$WORK instead would solve this problem (TACC creates this symbolic link and the variable when they create the account, so everybody should have it).

  7. Erik Schnetter
    • removed comment

    The Simfactory syntax for accessing environment variables is @ENV(HOME)@. Roland, can you try whether this works instead of \$HOME?

  8. Roland Haas
    • removed comment

    using @ENV(HOME)@ does not seem to work (see comment 4 I guess). I get:

    [rhaas@horizon Cactus]$ PYSIM_2010/bin/sim sync lonestar /bin/bash: -c: line 0: syntax error near unexpected token `(' /bin/bash: -c: line 0: `{ :; } && { cd '/home/rhaas/Cactus' && { :; } && ssh -Y rhaas@lonestar.tacc.utexas.edu mkdir -p @ENV(HOME)@/work/Cactus && rsync --rsh='ssh -Y' --rsync-path='/home1/01385/barrywar/software/rsync/bin/rsync' --archive --hard-links --sparse --verbose --progress --partial --delete --stats --compress --filter 'merge /home/rhaas/Cactus/PYSIM_2010/etc/filter.rules' par CONTRIBUTORS COPYRIGHT Makefile arrangements lib manifest repos src utils rhaas@lonestar.tacc.utexas.edu:@ENV(HOME)@/work/Cactus; }' /bin/bash: -c: line 0: syntax error near unexpected token `(' /bin/bash: -c: line 0: `{ :; } && { cd '/home/rhaas/Cactus' && { :; } && ssh -Y rhaas@lonestar.tacc.utexas.edu mkdir -p @ENV(HOME)@/work/Cactus && rsync --rsh='ssh -Y' --rsync-path='/home1/01385/barrywar/software/rsync/bin/rsync' --archive --hard-links --sparse --verbose --progress --partial --delete --stats --compress --filter 'merge /home/rhaas/Cactus/PYSIM_2010/etc/filter.prersync3.rules' par CONTRIBUTORS COPYRIGHT Makefile arrangements lib manifest repos src utils rhaas@lonestar.tacc.utexas.edu:@ENV(HOME)@/work/Cactus; }'

    Error while syncing to lonestar.

    Errors occurred during synchronisation.

    \$HOME/work on the other hand works fine (at least for sync).

  9. Frank Löffler
    • removed comment

    Please review the attached patch. It adds at least support for the @ENV()@ syntax for the sync command. I didn't test anything else, but according to Erik it's the right syntax to use. It works now for ranger to set sourcebasedir=@ENV(WORK)@

  10. Erik Schnetter
    • removed comment

    How does this compare to ParseEnvCommands? Should this routine now be deleted? (I just grepped for ENV in the source code.)

    This is only a partial implementation, since it only works for the sync destination. The general case still needs to be implemented; and this will need to happen in such a way that the special case for sync still works.

    Do we actually like the ENV() syntax? I though this was the syntax already implemented; since this is something new, e.g. ENV[] may also work, or ENVIRON[], or requiring quoting the argument...

  11. Frank Löffler
    • removed comment

    This is reported in ticket:595. In case it will be fully implemented, it should then be used within the sync code.

    I don't care much about the ENV syntax. The only possible source of confusion might be that Cactus parameter files use $ENV{'NAME'} as syntax. We could as well adapt simfactory to that. But then, simfactory already uses the @@ syntax in other places.

    My only concern is that we should have something working for the release. But that would require not only sync to work, because directories like SCRATCH should be used for job-creation as well. I did not test ENV(NAME) there yet, but given that ParseEnvCommands doesn't seem to replace everything with '' I have doubts that it works.

  12. Erik Schnetter
    • removed comment

    I suggest to use the simplest possible fix at the moment, which is to use \$HOME in the mdb entries. Anything else can be implemented after the release, when we have more freedom to experiment with syntax and functionality.

  13. Ian Hinder reporter
    • removed comment

    Using \$HOME will not solve the problem of the simulation directories, and so the user will have to customise anyway. Since we have just over a week to the release, I propose that we document what is needed for these machines in the release notes and in wherever people go to find out peculiarities of using certain machines with the ET (maybe we need such a place?). After the release, we can solve the problem properly by making a consistent environment variable syntax work everywhere in simfactory.

  14. Ian Hinder reporter
    • removed comment

    I have updated the [https://docs.einsteintoolkit.org/et-docs/Detailed_Release_Announcement release notes] to include this instruction for LoneStar, Ranger and Kraken, which are the three machines where I have run into this problem. If there is agreement that we can release in this way (the same problem was in the last release, after all), then we can remove the release milestone from this ticket and leave the ticket open for fix in the next cycle.

  15. Ian Hinder reporter
    • removed milestone
    • removed comment

    I am removing the release milestone and leaving the ticket open for a future fix.

  18. Ian Hinder reporter
    • removed comment

    As far as I know, the problem has not been solved, so the ticket should remain open.

  21. Log in to comment
Assignee
anonymous
Type
bug
Priority
minor
Status
closed
Component
SimFactory
Milestone
–
Version
–
Votes
0
Watchers
0
Jira Software: the preferred issue tracker for Bitbucket. Join the team!