NaNChecker should not reduce in local mode

Issue #765 closed
Erik Schnetter created an issue

The NaNChecker calls reduction operations in local mode. This does not quite work if a process has more than one component of the grid, and deadlocks if processes have different numbers of components.

Keyword:

Comments (10)

  1. Frank Löffler
    • removed comment

    I use it for simulations on one MPI processor but multiple levels and components there. What is the symptom I should see in that case? False positives?

  2. Erik Schnetter reporter
    • removed comment

    In your case (same number of components on each process), the bit mask seems to be reset for each component.

  3. Erik Schnetter reporter
    • removed comment

    If there are nans, the NaNChecker will find some nans. However, the bit mask in the output nanmask may be inconsistent.

  4. Roland Haas
    • removed comment

    The attached patch splits NaNCheckers operation into a local search routine followed by a per-level reduction. It does not use ENTER_LEVEL_MODE to not rely on Carpet's code (ie. to compile without Carpet present). Note that NaNChecker does not trigger the warning in CarpetReduce since it uses reducelocalscalar so does not reduce a grid function (it seems to end up in ReduceArrays inside of CarpetReduce). I attach a test for NaNChecker (using PUGH, which is likely fragily since it relies on IEEE floating point math to create NaNs on the grid).

  5. Roland Haas
    • changed status to resolved
    • removed comment

    I knew there was something that I had outstanding to commit :-) Though I had thought it was the change to IOJpeg. Applied.

  6. Log in to comment