Opened 7 years ago

Last modified 6 years ago

#620 reopened enhancement

Simplify timelevel handling for Cactus thorn writers

Reported by: Ian Hinder Owned by:
Priority: major Milestone:
Component: Cactus Version:
Keywords: Cc:


Currently, a thorn writer must declare the maximum number of timelevels for a variable in the interface.ccl file, then allocate a specific number of timelevels for which storage is allocated in the schedule.ccl file. There are rules for how many timelevels a variable should have. For example, to evolve using MoL I think you need at least two, whereas to evolve using mesh refinement with time prolongation order 2 you need three.

The problem is that the person writing the thorn shouldn't have to know how it is going to be used. If I write an evolution thorn, I should not care whether it is used with mesh refinement or not. I certainly shouldn't have to care what prolongation order is going to be used.

Would it be possible for Cactus to automatically determine the number of timelevels needed for a given variable? My proposal is that it should not be necessary for the user to specify the number of timelevels in the interface.ccl or schedule.ccl file for variables declared in that thorn. The only time a thorn writer should have to do that is if that thorn specifically uses the other timelevels. For example, a thorn which couples to MoL will probably only ever read or write to the current timelevel. MoL, on the other hand, could tell Cactus that it needs a certain number of timelevels at runtime for those variables. Similarly for mesh refinement.

This goes along with the planned changes to the scheduling system to make it easier to program Cactus thorns and make it harder to make errors.

Attachments (0)

Change History (8)

comment:1 Changed 7 years ago by Erik Schnetter

Each timelevel is stored in its own variable, e.g. gxx, gxx_p, gxx_p_p, etc. The number of timelevels declared in interface.ccl defines how many of these variables will be passed into each subroutine, and thus needs to be known at build time.

The number of timelevels that is allocated is much more flexible; this is the number specified in the schedule.ccl file.

There is, in principle, nothing wrong with allocating more timelevels than there are variables. The additional timelevels could not be accessed in "straightforward" code, but are still visible e.g. via CCTK_VarDataPtr.

That is, it should be fairly easy to do away with the maximum number of timelevels. The number of timelevels declared in the interface.ccl becomes the number of timelevels directly accessible in the code -- typically only 1 is necessary, except if past timelvels are initialised explicitly, which is rarely a convenient way to do so.

comment:2 Changed 7 years ago by Erik Schnetter

I believe this could be implemented in the following way:

  1. In addition to the flesh function MaxTimelevels that returns the maximum number of timelevels, we add a SetMaxTimelevels that allows increasing the maximum number of timelevels for a group. We probably don't want to allow decreasing this number, because some code may cache this number (maybe implicitly in the CCTK_ARGUMENTS bindings). This would not actually allocate memory, it only increases the maximum. The flesh provides a data structure cGH->data which contains pointers to all timelevels of all grid variables; this data structure needs to be resized (and the new elements initialised) in this function.
  1. The driver functions that modify the number of active timelevels call SetMaxTimelevels if necessary before they allocate storage. If a driver does not do this, the behaviour remains as is at the moment.
  1. The drivers that allow increasing this number will also have to checkpoint and restore this value while checkpointing. It seems that the checkpointing data format does not need to be changed; increasing the maximum to the number of timelevels found in the checkpoint file seems fine.

This data structure is set up in CactusDefaultSetupGH (file Comm/CactusDefaultComm.c).

The maximum number of timelevels per group is stored in the field n_timelevels of struct cGroupDefinition, which is declared privately in main/Groups.c, and can be read via MaxTimelevels.

Does this sound reasonable?

comment:3 Changed 7 years ago by Erik Schnetter

Status: newreview

comment:4 Changed 6 years ago by Frank Löffler

It does.

comment:5 Changed 6 years ago by Erik Schnetter

Status: reviewreopened

This is a good idea. Setting status to "open" since there is no patch yet.

comment:6 Changed 6 years ago by Ian Hinder

That would address interface.ccl. What about schedule.ccl? I would like an application thorn to only have to declare the number of timelevels that it actually needs. So a wave equation thorn, writing to only one timelevel, would just say

  STORAGE: phi[1]

I think the 1 is actually optional in this case. Then we need a way for MoL to dynamically tell the driver that it needs two timelevels for phi, and Carpet to indicate that it needs 3 if the time prolongation order requires it. What would be a good way of implementing this? It is not quite the same as SetMaxTimelevels.

comment:7 Changed 6 years ago by Ian Hinder

There is already a flesh function CCTK_GroupStorageIncrease ( Maybe we can just add calls to this function to MoL and Carpet to increase the amount of storage when it is needed? This function would then call SetMaxTimelevels to ensure that the data structures had space for enough timelevels, as described above.

comment:8 Changed 6 years ago by Erik Schnetter

The current mechanism for a thorn indicating that it needs time levels is to call cctk_GroupStorageIncrease, as this would allocate new time levels if necessary. This call will never reduce the number of time levels.

These calls need to occur late enough, so that the GH already exists and grid functions can have storage allocated, but also needs to occur early enough that initial data can still be set up. I believe basegrid would be a good time.

Modify Ticket

Change Properties
Set your email in Preferences
as reopened The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
The owner will be changed from (none) to anonymous.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.