move PARAMCHECK before WRAGH

Issue #1448 new
Roland Haas created an issue

currently PARAMCHECK runs after WRAGH which means that the lev0 data and grid arrays as well as grid scalars have storage (but the grid is not yet set up properly since basegrid did not yet run).

This means that memory to hold level 0 has been allocated which can be quite large (eg for mulitpach runs where level 0 encompases all multipatch grids). Therefore using cactus_sim -P parfile.par to test parameters and parameter values before submitting a job (which is good practise) can potentially cripple the head node on which one tries this.

Would it be possible to to swap the order (and make sure to call CCTKi_FinaliseParamWarn right after PARAMCHECK) or is do we require that storage exists during PARAMCHECK?

For the code that provides the current ordering, see CallSetup in Carpet/src/Initialise.cc and src/main/CactusDefaultInitialise.c (in the flesh).

Keyword:

Comments (5)

  1. Erik Schnetter
    • removed comment

    This may work. You would want to move "CallSetup" to before "CCTKi_ScheduleGHInit". Worth a try...

  2. Roland Haas reporter
    • removed comment

    Not sure I follow. I want to only change the order fo WRAGH and PARAMCHECK both of which are traversed from within CallSetup (lines 183 and 186):

    182      // Register coordinates
    183      ScheduleTraverse (where, "CCTK_WRAGH", cctkGH);
    184      
    185      // Check parameters
    186      ScheduleTraverse (where, "CCTK_PARAMCHECK", cctkGH);
    187    } END_MGLEVEL_LOOP;
    

    I have not tried this, but wouldn't moving WRAG before ScheduleGHInit by a bad idea?

  3. Erik Schnetter
    • removed comment

    WRAGH is supposed to be before PARAMCHECK, so that one can set up certain data structures that may be required for parameter checking later on.

    However, you are right that the grid structure should not be allocated before PARAMCHECK. Thus moving all the routines that handle the grid structure setup, schedule initialization, etc. should be called after PARAMCHECK. This may create some fall-out and/or may require some changes to schedule routines, but it should be possible.

  4. Roland Haas reporter
    • removed comment

    ok. This will require some rewriting of code inside of Carpet I think. Right now the docs for the time bins ion page D28 of the user guide) say:

    CCTK WRAGH This timebin is executed when all parameters are known, but before the driver
              thorn constructs the grid. It should only be used to set up information that is
             needed by the driver.
    CCTK PARAMCHECK This timebin is for thorns to check the validity of parameter combinations. This
                   bin is also executed before the grid hierarchy is made, so that routines scheduled
                  here only have access to the global grid size and the parameters.
    

    so no one in PARAMCHECK may expect access to grid functions and technically grid arrays and grid scalars are also not mentioned. Checking the various schedule.ccl there are routines scheduled in WRAGH (eg CarpetRegrid2_Initialise) that access grid scalars. So we will need to turn on grid arrays and grid scalars before WRAGH (and hope that those are "small" which I have at least one thorn for which this is not true since it allocates 3d distributed grid arrays). Probably best committed after the release.

  5. Erik Schnetter
    • removed comment

    Carpet does internally not distinguish between grid functions and grid arrays. There is a distinction between variables with global storage (defined in top-level statements in the scheduler) and variables with temporal storage (defined in schedule groups or scheduled routines). From this point of view, it would be cleaner to not allocate any storage before paramcheck.

    CarpetRegrid2 could run later, early in the basegrid bin.

    Instead of moving things from wragh to basegrid, we could introduce a new bin "basebasegrid" that is run before basegrid, but only in global mode. This may help move schedule entries around.

  6. Log in to comment