1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

allow new timelevels to be created at runtime

Issue #1797 closed
Roland Haas created an issue

The set of patches in

https://bitbucket.org/cactuscode/cactus/branch/rhaas%2FunlimitedTimelevels https://bitbucket.org/cactuscode/cactuspugh/branch/rhaas%2FunlimitedTimelevels https://bitbucket.org/eschnett/carpet/branch/rhaas%2FunlimitedTimelevels

extend the flesh and the PUGH and Carpet drivers such that user thorns can enable more timelevels than the TIMELEVELS attribute in interface.ccl specifies.

This separates the number of _p and _p_p etc. variables, which continues to be controlled by interface.ccl's TIMELEVELS attribute, from the number of timelevels that are accessible via CCTK_VarDataPtr().

This means that evolution thorns can ask for only ask many timelevels as they need for their own evolution algorithm (so 2 of MoL based thorns) and don't have to worry about how many timelevels Carpet may require for time interpolation. Similarly some MoL evolution methods (for example AB type methods) require multiple past timelevels of the RHS variables which MoL can enable by its own when using this functionality.

Their should be very little user visible changes, mostly since few thorns actually needed to know the number of timelevels. The exisiting flesh functions CCTK_MaxTimeLevels continue to return the value set in TIMELEVELS while new functions CCTK_MaxActiveTimeLevels return the maximum number of timelevels that were ever activated or CCTK_MaxTimeLevels, whichever is larger.

A possible improvement would be to make the existing CCTK_MaxTimeLevels return what CCTK_MaxActiveTimeLevels, yet no user code should ever need knowledge of CCTK_MaxTimeLevels anyway since it only ever returned the value set in interface.ccl so only drivers etc. should have used it to check for invalid timelevel parameters.

The downside of that approach is that it changes the meaning of an existing flesh function rather than introducing a new one and deprecating the old one. This is fine if the meaning of CCTK_MaxTimeLevels really is "upper bound on CCTK_ActiveTimeLevels that could have been seen" rather than "the value of TIMELEVELS from interface.ccl".

The patches pass all tests.

Keyword: Cactus
Keyword: Carpet
Keyword: PUGH

Comments (12)

  2. Erik Schnetter
    • removed comment

    What happens during time level cycling? Currently, grid variables with one time level remain unchanged, whereas variables with more than one time level become invalid during time level cycling.

    I suggest to change this behaviour so that every grid function remains defined at its current value during time level cycling. This will then avoid the need to copy the past to the current time level, which is the first step of every time integrator.

  3. Roland Haas reporter
    • removed comment

    Replying to [comment:2 eschnett]:

    What happens during time level cycling? Currently, grid variables with one time level remain unchanged, whereas variables with more than one time level become invalid during time level cycling. The patches introduce no new behaviour. Maybe I was not clear enough: What the proposed patches actually do is make the flesh ignore the TIMELEVELS attribute in interface.ccl in the EnableStorage calls.

    The described issue of a variable going from not changing to changing already happens currently with eg a TIMELEVELS=2 variable whose schedule.ccl says STORAGE: foo[1] and where the user later calls EnableStorage during runtime.

    I suggest to change this behaviour so that every grid function remains defined at its current value during time level cycling. This will then avoid the need to copy the past to the current time level, which is the first step of every time integrator. This would basically amount to moving the InitialCopy routine out of MoL and into the driver, just after it cycled timelevels, wouldn't it? Or are you suggesting a more invasive change that couples the time stepper and the driver more tightly?

  4. Erik Schnetter
    • removed comment

    The case I'm worried about is the following: Assume there's some code that allocates only a single time level. Now some other thorn "secretly" increases the number of time levels, e.g. for time interpolation. This will then "secretly" destroy the current time level during time level cycling.

    I propose to modify PUGH to copy the past to the current time level, and to modify Carpet to not overwrite the current timelevel with nans. MoL's copy routine would then be omitted.

  5. Ian Hinder
    • removed comment

    So timelevel cycling would then have the effect of copying tl_N into tl_{N+1}, and leaving tl0 as it is, so that it is identical to tl1? This sounds like a good idea to me. If we can then omit MoL's copy, I think we can then use MoL with only a single timelevel with, e.g. RK4, which is all it should need.

  6. Roland Haas reporter
    • removed comment

    (Answering only Erik so far).

    I see, so I can see that currently a thorn can protect against this type of thing if it marks a grid function as TIMELEVELS=1.

    I am not sure though if I can construct a situation where this would make copying forward just after cycling the right thing to do: I feel that if time cycling happens because multiple timelevels exist, then the grid function is now time dependent (since all reasons for having time levels means that something changes with time). In this case the copied forward data is actually wrong since it is not the data that belongs to cctk_time but instead that that belongs to cctk_time - cctk_delta_time (since tl=0 is the invalid level and tl=1 is the valid one that was just cycled out). This is fine in MoL's evolution loop since MoL basically says "compute f(y, t) for the y and t that you find in tl=0 and cctk_time" and itself modifies cctk_time so that for eg the first RK step cctk_time is reduced by cctk_delta_time compared to what the driver supplied (in MoL/src/SetTime.c). However the data in tl=0 is genuinely invalid between the time Carpet cycled it in and the time MoL finishes (so in particular the PRESTEP and the PREREGRID bins).

    Note that the copied data is correct (in that is is what you'd expect) for something like the ADM variables if they are copied forward at the same time as the BSSN variables only in the sense in that the ADM variables are what would be computed by a BSSN2ADM routine given the values of the BSSN variables in tl=0. I would however maintain that those values (of BSSN in tl=0) are incorrect in between time cycling and the end of EVOL.

    Well actually, I can think of a situation, namely say the RHS grid functions which usually have only a single timelevel. A thorn could currently rely on RHS not changing between the last RK step and the first one of the next iteration and use this to compute the new RHS. My proposed solution to this would be for MoL to handle RHS grid functions the same way it does constrained ones: copy forward in time if they have more than one timelevel. Was that the situation you had in mind?

  7. Roland Haas reporter
    • removed comment

    Replying to [comment:5 hinder]:

    So timelevel cycling would then have the effect of copying tl_N into tl_{N+1}, and leaving tl0 as it is, so that it is identical to tl1? This sounds like a good idea to me. If we can then omit MoL's copy, I think we can then use MoL with only a single timelevel with, e.g. RK4, which is all it should need.

    Avoiding the copying does not avoid multiple timelevels since MoL uses tl=0 as scratch space during the computation. Eg the final LinearCombination for RK4 is:

    y_0_new = 1.0 * y_0_from_last_RK_step + scratchVar - 4/3 * y_1

    What Erik proposes does not change anything from the point of the MoL time intergrator, it just moves MoL's Initialcopy from the beginning of EVOL (when MoL's group runs) to just before CCTK_PRESTEP (when Carpet cycles timelevels).

  8. Roland Haas reporter
    • removed comment

    As discussed in the ET call two weeks ago, I removed the CCTK_MaxTimeLevels function from the flesh and added code to disallow increasing the number of active timelevels beyond what is declared in interface.ccl unless at least 2 timelevels are declared. Changed thorns:

  9. Roland Haas reporter
    • removed comment

    For convenience I attach a thornlist to check out the required branches. Please review.

  13. Log in to comment
Assignee
–
Type
enhancement
Priority
minor
Status
closed
Component
Cactus
Milestone
–
Version
development version
Votes
0
Watchers
0
Jira Software: the preferred issue tracker for Bitbucket. Join the team!