- changed status to open
- removed comment
Carpet with Periodic Boundary Conditions
Issue #558
closed
The current Carpet driver is not compatible with the periodic boundary conditions (as provided by the Periodic thorn or otherwise).
Erik's comment on the User's list was: "It is straightforward to implement periodicity in the grid structure -- one needs to take the current grid structure, shift it in the 26 directions, and take the logical union of all 27 grid structures. This will then be automatically clipped. The most complex part is calculating by how much to shift."
Please implement support for periodic BCs in Carpet. This will directly enable production science runs.
Keyword: carpet
Keyword: periodic
Comments (26)
-
-
reporter
- removed comment
Unfortunately, that does not seem to work. I tried setting: CarpetRegrid2::symmetry_periodic_x = yes CarpetRegrid2::symmetry_periodic_y = yes CarpetRegrid2::symmetry_periodic_z = yes and with and without Periodic::periodic = yes. When the regridding meets the boundary, odd values appear.
-
reporter
- removed comment
It looks like something is wrong with the testing condition, because even when the regridding meets the boundary, I never see the "making regions periodic in the..." message with veryverbose turned on.
-
- removed comment
I believe the offset (integer domain size) is calculated wrong. I attach a new patch.
-
reporter
- removed comment
Unfortunately, the new patch also does not seem to work. Also with this version, I never see the "making regions periodic in the..." message with veryverbose turned on.
I'll post my stdout.log file.
-
- removed comment
Next attempt; the offset was calculated wrong.
-
reporter
- removed comment
We seem to be getting closer. Do we need to insert a boundary_clip() operation after the fixup in each direction? I ask because... Refinement level 1: making regions periodic in the x direction... New regions are bboxset<CCTK_INT4,3>:{size=164454,setsize=63,set=[([... Refinement level 1: making regions periodic in the y direction... New regions are bboxset<CCTK_INT4,3>:{size=493362,setsize=189,set=[([
and it is still running after many hours, seemingly stuck while executing: CarpetRegrid2::periodic<1>::test_impl(...), specifically executing the line: ibset const difference = symmetrised - regions.AT(rl); (so this is in the post-enforcement check for the y direction)
-
reporter
- removed comment
It did eventually move on to doing CarpetRegrid2::periodic<2>::test_impl, so it was not in an endless loop, but it then started processing a box set with some 1.8k members.
If I understand the problem correctly, then inserting extra calls to: properties.push_back (new boundary_clip()); after the queueing of periodic<0>() and periodic<1>(), should help, but that does not seem like a general solution (because it implies extra overhead even when the periodic b.c. handing is turned off).
-
- removed comment
Inserting an extra call to clip should not be a performance issue, because these routines are only called during regridding, not during evolution.
One alternative may be to modify the shifting to
Shift boxes upwards and downwards by one period symmetrised |= bb.shift(+ioffset / bb.stride()) & baseextent; symmetrised |= bb.shift(-ioffset / bb.stride()) & baseextent;
which does part of what the clipping does.
The underlying problem, of course, is Carpet's inefficient implementation of bboxsets. This really needs to improve.
-
reporter
- removed comment
Inserting the extra calls to: properties.push_back (new boundary_clip()); does allow the regridding to proceed in a reasonable amount of time, but, unfortunately, something is still wrong. I've attached new image from right after it regrids on the boundary. As you can see, the grid now does seem periodic, but bogus data seems nevertheless to be copied into the level 1 cells.
-
- changed status to open
- removed comment
As long as something is wrong with the patch, it should probably not be up for review.
-
- removed comment
I would also be interested in getting this to work. I've tried to reproduce the problem using a slight variant of ks-mclachlan.par (attached). It seems that the problem is not connected to the new patch, which as far as I understand seems correct, but to some other problem in Carpet which emerges when the refinements are touching the outer boundaries. In brief, some boundary points are poisoned starting from the first iteration.
-
- removed comment
I tried this parameter file, and I did not see any nans. How many processes do you use? Which points of which variables contain poison at what iteration?
I do see a problem with the reduction weight that seems to be set up wrong.
-
- removed comment
I've attached a parameter file that uses cell centering instead of vertex centering. This is more natural for periodic boundaries, because upper and lower boundary can be treated in the same way. Can you try this parameter file?
-
- removed comment
I see the same behavior with the CC parfile. I'm running with one process, one thread.
-
- removed comment
- changed watchers to eloisa
-
- removed comment
Eloisa, you mentioned poison. Poison is much easier to verify and debug than looking at png images. Can you tell me when and where you are seeing poison, and how many processes you use?
-
- removed comment
Indeed, it is poison. I've now realized that the issue with periodic boundaries is not the same as the one that arises on actual physical boundaries, reported above. I'm now concentrating on the issue with periodic boundaries: the poison is on all the boundary points of refinement levels with rl>0. I'm looking into thorn Periodic.
-
- removed comment
Specifically, it seems to be a problem with the Gammas, A, B, etc, (which are initialised using finite differencing) on the UPPER intersection of the symmetry and refinement boundary. This seems very similar to what happens with multipatch, where the intersection of the outer and interpatch (symmetry) boundary is not initialised. We said this was due to the order in which things happen.
So here we are seeing problems with the intersection of refinement boundaries and either symmetry or outer boundaries (probably two separate issues), and with multipatch we see problems with the intersection of symmetry and outer boundaries, to complete the set!
-
- removed comment
I don't see poison nor nans in my runs. Can you attach an output file with that problem?
-
- removed comment
Please find the 1D output of ml_gamma below. Please notice that the poison can occur both on the upper and on the lower faces; I've slightly modified the parameter file to show that.
-
- removed comment
Erik, could you give me a location where I can find the Formaline tars for your setup? Maybe this is the fastest way to figure out what's different.
-
- removed comment
I've looked into this in more depth and I've found at least two major issues that prevent the original setup from working correctly:
1) Slab (used by Periodic for filling the symmetry boundaries) doesn't seem to work when moving data between two components of the grid that are owned by the same process (this is also stated in the documentation); as a result, when a refinement level is split into two components (as would have to be the case for one that overflows on one side of the domain and has a chuck appearing on the opposite face) and they are both handled by the same process, the boundaries are simply not filled. If these points existed when the initial data was set, they keep those values forever. If the refinement appears later, it will contain poison. This problem can be reproduced with ks-mclachlan.par, and I'll create a new ticket for Slab; 2) The original example also tries using true AMR. I'm quite unfamiliar on how this should be done, and would like to understand exactly at which point the level_mask needs to be set. With the current location (preregrid), the grid structure will be different depending on whether poisoning is used, which seems to point out that something is used before the user thorn has the chance to set it. Notice that the question is complicated by the fact that the level_mask is set from a derivative of a grid function, so that it will take a few operations (interior loop, syncs, boundary conditions, …) to fill the whole domain with the correct values. The ET list is probably a good place for this discussion.
Regarding the original request of making Carpet aware of periodic boundaries, as far as I can test Erik's third patch works and closes this ticket.
-
- changed status to resolved
- removed comment
ad 1: Yes, this is correct. This used to not be a problem (in 2003), and comes back to bite now. Apologies for not implementing a test for this -- all other symmetry thorns do have such a test. There is now a ticket for this,
#694.ad 2: PreRegrid is the place. Let's discuss on the list.
Closing the ticket since the original issue seems to be solved.
-
- removed comment
Applied.
-
- changed status to closed
- edited description
- Log in to comment
Please try the attached patch.