Opened 5 years ago

Last modified 5 years ago

#1448 new defect


Reported by: Roland Haas Owned by:
Priority: minor Milestone:
Component: Cactus Version: development version
Keywords: Cc:


currently PARAMCHECK runs after WRAGH which means that the lev0 data and grid arrays as well as grid scalars have storage (but the grid is not yet set up properly since basegrid did not yet run).

This means that memory to hold level 0 has been allocated which can be quite large (eg for mulitpach runs where level 0 encompases all multipatch grids). Therefore using cactus_sim -P parfile.par to test parameters and parameter values before submitting a job (which is good practise) can potentially cripple the head node on which one tries this.

Would it be possible to to swap the order (and make sure to call CCTKi_FinaliseParamWarn right after PARAMCHECK) or is do we require that storage exists during PARAMCHECK?

For the code that provides the current ordering, see CallSetup in Carpet/src/ and src/main/CactusDefaultInitialise.c (in the flesh).

Attachments (0)

Change History (5)

comment:1 Changed 5 years ago by Erik Schnetter

This may work. You would want to move "CallSetup" to before "CCTKi_ScheduleGHInit". Worth a try...

comment:2 Changed 5 years ago by Roland Haas

Not sure I follow. I want to only change the order fo WRAGH and PARAMCHECK both of which are traversed from within CallSetup (lines 183 and 186):

182      // Register coordinates
183      ScheduleTraverse (where, "CCTK_WRAGH", cctkGH);
185      // Check parameters
186      ScheduleTraverse (where, "CCTK_PARAMCHECK", cctkGH);

I have not tried this, but wouldn't moving WRAG before ScheduleGHInit by a bad idea?

comment:3 Changed 5 years ago by Erik Schnetter

WRAGH is supposed to be before PARAMCHECK, so that one can set up certain data structures that may be required for parameter checking later on.

However, you are right that the grid structure should not be allocated before PARAMCHECK. Thus moving all the routines that handle the grid structure setup, schedule initialization, etc. should be called after PARAMCHECK. This may create some fall-out and/or may require some changes to schedule routines, but it should be possible.

comment:4 Changed 5 years ago by Roland Haas

ok. This will require some rewriting of code inside of Carpet I think. Right now the docs for the time bins ion page D28 of the user guide) say:

CCTK WRAGH This timebin is executed when all parameters are known, but before the driver
          thorn constructs the grid. It should only be used to set up information that is
         needed by the driver.
CCTK PARAMCHECK This timebin is for thorns to check the validity of parameter combinations. This
               bin is also executed before the grid hierarchy is made, so that routines scheduled
              here only have access to the global grid size and the parameters.

so no one in PARAMCHECK may expect access to grid functions and technically grid arrays and grid scalars are also not mentioned. Checking the various schedule.ccl there are routines scheduled in WRAGH (eg CarpetRegrid2_Initialise) that access grid scalars. So we will need to turn on grid arrays and grid scalars before WRAGH (and hope that those are "small" which I have at least one thorn for which this is not true since it allocates 3d distributed grid arrays). Probably best committed after the release.

comment:5 Changed 5 years ago by Erik Schnetter

Carpet does internally not distinguish between grid functions and grid arrays. There is a distinction between variables with global storage (defined in top-level statements in the scheduler) and variables with temporal storage (defined in schedule groups or scheduled routines). From this point of view, it would be cleaner to not allocate any storage before paramcheck.

CarpetRegrid2 could run later, early in the basegrid bin.

Instead of moving things from wragh to basegrid, we could introduce a new bin "basebasegrid" that is run before basegrid, but only in global mode. This may help move schedule entries around.

Modify Ticket

Change Properties
Set your email in Preferences
as new The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
Next status will be 'confirmed'.
The owner will be changed from (none) to anonymous.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.