add tags to disallow splitting in directions

Issue #1445 closed
Roland Haas created an issue

the attached patch adds a new tag "no_split_directions" which can be used to prevent a grid array to be split in the given directions when processor decompositions is done. I currently explicitly forbid this tag to be used in grid functions since all grid functions share the same processor decomposition so using this tag on a per-function basis is impossible.

Keyword:

Comments (5)

  1. Erik Schnetter
    • changed status to open
    • removed comment

    The statement

     bvect const dims = not split_components || no_split_dims;
    

    seems to be the pivot point of the whole implementation. I assume you tested it... too lazy to figure out whether this is correct.

    Please apply.

  2. Roland Haas reporter
    • changed status to resolved
    • removed comment

    Hmm. We have tested that the tag works with grid arrays (in the thorn for which this functionality was introduced, "Replay" in Zelmani if you are curious). I had not tested whether I had broken split_components. Let me see.

    Hmm, alright, I am tempted to claim that Carpet::split_components = "no" is not usable right now. Setting it triggers an assert(nprocs == 1) in SplitRegionsMaps_Automatic_Recursively unless CCTK_NProcs() == 1 in which case splitting components makes no sense anyway (since all(dims) == true). dims is set to all true just before we enter SplitRegionsMaps_Automatic_Recursively (namely in line 1715):

        for (int r=0, p=0; r<nregs; p+=mynprocs.AT(r), ++r) {
          if (recompose_verbose) cout << "SRMA superreg[" << r << "] " << superregs.AT(r) << endl;
          // bvect const dims = false;
          bvect const dims = not split_components || no_split_dims;
          SplitRegionsMaps_Automatic_Recursively
            (dims, p, mynprocs.AT(r), superregs.AT(r), newregs);
        } // for r
    

    I am no sure how to construct a parameter file to make this pass. Anyhow, I did not change its behaviour it seems :-).

    I'll revive CarpetTest and make it use the tag for one of its arrays.

    Committed as git hash 911ab59919a935b1d25146830673b2d200c74fed "Carpet: add new group tag no_split_directions".

  3. Roland Haas reporter
    • removed comment

    Hmpf. There was a bug left of course. Just such that for the choice of parameters in my original application it was producing the correct result. Fixed and added a test case to CarpetTest. Turns out that

    for(int i = 0 ; i < nvals . ++i) {
      no_split_dims[i] = true;
    }
    

    is identical to

    for(int i = 0 ; i < nvals . ++i) {
      no_split_dims[no_split_directions[i]] = true;
    }
    

    if no_split_directions = {0,1}.

  4. Log in to comment