1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

support make parallel options in simfactory

Issue #1928 closed
Roland Haas created an issue

Pull request https://bitbucket.org/simfactory/simfactory2/pull-requests/11/parallel-make/diff adds a --jobs option to sim build that sets a @MAKEJOBS@ replacement so that one can use make -j @MAKEJOBS@ in machine definitons files and let the user choose the number of make jobs when compiling

Keyword:

Comments (16)

  2. Erik Schnetter
    • removed comment

    Currently, Simfactory's machine database contains the preferred number of make jobs. Do you want to override this manually on a case-by-case basis? Is there a case where the default settings are inconvenient?

  3. Frank Löffler
    • removed comment

    I could imagine that, from time to time, another than the default number of cores might be of interest. Say, if a build isn't time critical, and shouldn't take a large portion of a machine's cores. I didn't look at the patch yet, but I think the request is understandable. I would go as far as to see if a short "-j" option would be possible.

  4. Roland Haas reporter
    • removed comment

    Right now nothing changes since none of the ini files use @MAKEJOBS@. One can add makejobs to the ini files and use this as the default which can be overridden by the user. Eg.

    [zwicky]
makejobs = 6
make = make -j @MAKEJOBS@

    which can then be overridden by the user on the command line (not tested yet).

    The defaults are often inconvenient on the OSX and generic Linux builds (since one cannot now in advance whether the build is a raspberry pi or a 16 core corei7 workstation), see https://bitbucket.org/simfactory/simfactory2/commits/e6784d678365/ . I occasionally use a different number of make jobs on clusters but mostly for debugging or compile time error fixes when I want "-j1" so that make stops right after the error and error messages are not mixed with regular output from other make jobs.

  5. Roland Haas reporter
    • removed comment

    Replying to [comment:3 knarf]:

    I could imagine that, from time to time, another than the default number of cores might be of interest. Say, if a build isn't time critical, and shouldn't take a large portion of a machine's cores. I didn't look at the patch yet, but I think the request is understandable. I would go as far as to see if a short "-j" option would be possible.

    -j is already implemented.

  6. Frank Löffler
    • removed comment

    If I read the code correctly, '-j' (without argument) would imply '-j 1'. Is this correct? If so, is this intended? Usually it means '-j infinity' (for make).

  8. Erik Schnetter
    • removed comment

    If I don't specify an option, then Simfactory should use a system-specific default, not 1.

  9. Roland Haas reporter
    • removed comment

    Sure. The system specific default must be set in the machine.ini file for the machine by specifying a value of makejobs (same as the system specific default number of threads when running the simulation). If none is specified (and make is actually "make -j @MAKEJOBS@" rather than "make -j 12" or so) then @MAKEJOBS@ defaults to 1 unless either overridden by a setting in the machine.ini file or by a -j or --jobs option on the command line.

    Obviously if you just hard-code "make" as "make -j 12" in the machine.ini file then no value for --jobs will do anything.

  10. Roland Haas reporter
    • changed status to open
    • removed comment

    Replying to [comment:9 rhaas]:

    Sure. The system specific default must be set in the machine.ini file for the machine by specifying a value of makejobs (same as the system specific default number of threads when running the simulation). If none is specified (and make is actually "make -j @MAKEJOBS@" rather than "make -j 12" or so) then @MAKEJOBS@ defaults to 1 unless either overridden by a setting in the machine.ini file or by a -j or --jobs option on the command line.

    Obviously if you just hard-code "make" as "make -j 12" in the machine.ini file then no value for --jobs will do anything.

    However it currently does not do this yet. Needs to be fixed.

  11. Frank Löffler
    • removed comment

    Replying to [comment:7 rhaas]:

    -j without an argument should abort with an error stating that -j requires an argument.

    That should be fine. Even 'make -j' (no number specified) is hardly ever useful.

  12. Roland Haas reporter
    • changed status to open
    • removed comment

    Now it does handle makejobs in ini files and --jobs on the command line properly I believe. This is in git hash 88f3ddb which is the new head of the pull request.

  13. Roland Haas reporter
    • removed comment

    This should still be reviewed. Note that I implemented the requested changes.

  17. Log in to comment
Assignee
–
Type
enhancement
Priority
minor
Status
closed
Component
SimFactory
Milestone
–
Version
development version
Votes
0
Watchers
0
Jira Software: the preferred issue tracker for Bitbucket. Join the team!