- removed comment
The number of timelevels for the AHFinder mask needs to be set to something intelligent.
Issue #380
resolved
The number of timelevels for AHFinderdirect::ahmask has to be set to something intelligent. Setting it to some fixed number doesn't work, because the number of timelevels for a given time-integrator might be different and Carpet chokes in that situation. At the moment it's set to how-many timelevels ADMBase::metric has, but that's also not good because that might be 1, while ahmask should have more. We cannot couple AHFinderDirect to any evolution thorn or Carpet as well. We might have to introduce a new parameter, but I would like to avoid this if somehow possible (which is why I coupled it to the metric in the first place, hoping this would be ok, which is apparently not really the case).
Keyword:
Comments (17)
-
-
- removed comment
So - would it be sufficient to change the parameter file, for the release?
-
- removed comment
I don't think that changing the scheduling is the correct approach here. Something wants to interpolate ahmask, and the ahmask isn't set up to allow this. Either ahmask needs 3 time levels, or it must not be interpolated (or it must be interpolated with a different operator).
-
- removed comment
I agree with Erik in that scheduling is not the solution (in fact it does not help anyway) for the apparent horizon finding during the actual evolution. There (for the run I ran) I reduced the finding frequency of AHFinderDirect to the coarsest time step (for all horizons, though maybe only the larger surfaces were required). This kind of corresponds to not interpolating ahmask.
Scheduling during ANALYSIS helps with Carpet::init_3_levels since Carpet calls CCTK_POSTSTEP during CCTK_INITIAL time when presumably cctk_iteration = 0 (but eg. cctk_time = -delta_t) so all of AHFinder's checks say "run me now". (I think from what I understand of how Carpet and the interpolation works.) I am not sure if I'd be happy with a full three-timelevel grid function just so that I can use it once during initial data. At least the my hydro runs tend to be starved for memory so I like avoiding extra grid functions (even though it is the cleanest solution to the problem). Maybe one coud make the number of timelevels a parameter?
One could avoid scheduling in ANALYSIS by using Carpet::fill_timelevels instead of init_3_levels.
-
- removed comment
If the mask is only needed during initialisation, and you are tight on memory, consider deallocating the mask after it is no longer needed.
fill_timelevels is the wrong approach for QC0, since this is not a stationary configuration. This will lead to a very large error initialliy. In fact, even init_3_timelevels is not good enough since it is only second order accurate.
This is a vacuum run; why is the ahmask necessary here at all?
-
- removed comment
AHFinder uses this grid function (which is zero outside of the horizon non-zero inside and in a buffer zone) to find out if it is using points from within the horizon. There is a also a noshrink option to AHFinder which prevents horizons from ever shrinking (and uses the mask to detect this). From looking at it (had to because this actually caused aborts for me at one point) this seems related to excision (expansion.cc line 928).
I don't think that the mask is required for anything else (in vacuum, it is very useful to have around for hydro).
The mask is used all the time not just during initialization so I cannot turn it off. I wanted to say that there are two issues: ahmask usage during evolution where you can set find_every = coarsest_step and the one during initialization where find_every will not help.
I am happy with whatever solution is decided upon. I just looked at AHFinder's schedule.ccl and it already allocates several timelevels based on a parameter: ADMBase::metric_timelevels So setting this to 3 should solve the issue in a nice way.
-
- removed comment
I think ahmask is used only for excision, and since we don't use excision in our vacuum simulations we don't need it. Without excision, horizons can shrink, and there is no reason to disallow using points from inside the horizon.
AHFinderDirect has a few parameters called use_mask, set_mask_* etc., and these should all be false in the sample QC0 simulations. Apparently they are not.
In the short term, I would follow your suggestion and increase the number of time levels, since this seems safer.
In the long run, I would like to add a facility to completely disable this mask. In our vacuum simulations, the mask is set to all zeros anyway.
-
- changed title to The number of timelevels for the AHFinder mask needs to be set to something intelligent.
- marked as
- changed milestone to ET_2011_11
- removed comment
We now set the number of timelevels for the metric to three, which solves the problem for now. I change the description to match the problem which is not really solved.
-
reporter
- removed milestone
- removed comment
No solution is forthcoming for this release - removing milestone.
-
- removed comment
Is this still an issue?
-
This still happens and can be triggered in qc0-mclachlan.par by changing
diff --git a/par/qc0-mclachlan.par b/par/qc0-mclachlan.par index fbb6742..3d3e633 100644 --- a/par/qc0-mclachlan.par +++ b/par/qc0-mclachlan.par @@ -114,9 +114,9 @@ CoordBase::zmin = 0.00 CoordBase::xmax = +120.00 CoordBase::ymax = +120.00 CoordBase::zmax = +120.00 -CoordBase::dx = 2.00 -CoordBase::dy = 2.00 -CoordBase::dz = 2.00 +CoordBase::dx = 2.00*1.5 +CoordBase::dy = 2.00*1.5 +CoordBase::dz = 2.00*1.5 CoordBase::boundary_size_x_lower = 3 CoordBase::boundary_size_y_lower = 3 @@ -262,7 +262,7 @@ ADMBase::initial_dtlapse = "zero" ADMBase::initial_dtshift = "zero" # needed for AHFinderDirect -ADMBase::metric_timelevels = 3 +ADMBase::metric_timelevels = 1 TwoPunctures::par_b = 1.168642873 TwoPunctures::par_m_plus = 0.453 @@ -270,6 +270,10 @@ TwoPunctures::par_m_minus = 0.453 TwoPunctures::par_P_plus [1] = +0.3331917498 TwoPunctures::par_P_minus[1] = -0.3331917498 +twopunctures::npoints_phi = 6 +twopunctures::npoints_A = 6 +twopunctures::npoints_B = 6 + #TODO# TwoPunctures::grid_setup_method = "evaluation" TwoPunctures::TP_epsilon = 1.0e-2 @@ -354,7 +358,7 @@ Multipole::l_max = 4 ActiveThorns = "AHFinderDirect" -AHFinderDirect::find_every = 128 +AHFinderDirect::find_every = 1 #28 AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES = nowhich fails after 1 step with:
ERROR from host fdea4908.spdns.org process 1 while executing schedule bin CCTK_POSTSTEP, routine AHFinderDirect::AHFinderDirect_find_horizons in thorn CarpetInterp, file /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetInterp/src/interp.cc:1514: -> Grid function "AHFINDERDIRECT::ahmask" has only 1 active time levels on refinement level 1; this is not enough for time interpolation cactus_sim: /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.The “best” solution seems to me to actually change
AHFinderDirect::ahmaskto haveTIMELEVELS=1ininterface.cclandSTORAGE: ahmaskinschedule.ccl. This indicates to Carpet that the grid function in question is constant in time and thus it will not complain about it when trying to interpolate it in time. This is a lie of course but justified I think. Reasons for this:- the grid function is a mask that is 0 on the outside of horizons, 1 inside based on
SpaceMask::emaskand that one has only a single timelevel and is reset byAHFinderDirectinPOSTREGRID. Before each horizon findahmaskis set based onemask. - both grid functions use
prolongation=”none”tags thus there is no interpolation in space or time during prolongation for either one AHFinderDirectdoes not find horizons in postregrid but does still use the last-found horizon to initializeemaskwhich sets it to the last found horizons, it does not copy this over toahmaskleavingahmaskunitialized after a regrid (!) on the past timelevels that are not set before horizon finding happens. This will cause warnings of the type “one or more points on the trial horizon surface point is/are in an excised region“ be produced ifmask_is_noshrinkis set (which is the only case in which the result from interpolatingahmaskis actually used for anything).- since with
mask_is_noshrinkthe valid mask on future timelevels is always smaller than the valid mask on past ones, ignoring past ones during interpolation does not affect the check for inside / outside the valid region
To demonstrate the “shrink” one can use the attached modified parameter file ahfinder-test.par
The pull request https://bitbucket.org/einsteintoolkit/einsteinanalysis/pull-requests/10/rhaas-singlelevel/diff reverts changes setting the number of timelevels of
ahmaskto a value greater than 1, which avoids it becoming undefined after a regrid in past timelevels and adds explicit InterpNumTimelevels tags to use a single timelevel for interpolation. - the grid function is a mask that is 0 on the outside of horizons, 1 inside based on
-
- attached ahfinder-test.par
-
- edited description
- changed status to open
Please review.
-
- changed milestone to ET_2020_04
-
assigned issue to
Roland Haas
- edited description
-
Unless objected I will apply this after 2020-01-03.
-
- changed status to resolved
Applied as git hash 914cb4b1 "AHFinderDirect: allow up to 3 timelevels for backwards compatibility" of einsteinanalysis
-
- changed milestone to ET_2020_05
- Log in to comment
- Assignee
- Type
- Priority
- Status
- resolved
- Component
- Other
- Milestone
- ET_2020_05
- Version
- –
- Votes
- 0
- Watchers
- 0
Ian, it is the same issue for initialization (tried to run qc0 yesterday and had the same problem). The same workaround could be applied, though it is more hairy to do so in this case since qc0 is an actual evolution. The root cause is init_3_timelevels which does a ScheduleTraverse (where, "CCTK_POSTSTEP", cctkGH); (in initialise_3tl_evolve_IIb). I found however that I would get this error even later on in the simulation. My guess would be that since AHFinderDirect looks for horizons every 128 iterations (and coarse_step = 256) it could be looking at points on refinement level 0 (r>64 maybe when looking for the surface with areal radius 50?).