Carpet/Carpet/test/64k2.par test fails on Jenkins test machine

Issue #2114 resolved
Ian Hinder created an issue

The test Carpet/Carpet/test/64k2.par exits with nonzero and reports:

INFO (Carpet):    There are 0 3-dimensional grid arrays in 0 groups
INFO (Carpet):    (The number of variables counts all time levels)
ERROR from host 7ce14e5707a0 process 0
  while executing schedule bin (none), routine (no thorn)::(no routine)
  in thorn CarpetLib, file /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/CarpetLib/mem.cc:113:
  -> Failed to allocate 12884901888 bytes (12884.902 MB) of memory for type CCTK_REAL8.  312 bytes (0.000 MB) are currently allocated in 39 objects
ERROR from host 7ce14e5707a0 process 0
  while executing schedule bin (none), routine (no thorn)::(no routine)
  in thorn CarpetLib, file /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/CarpetLib/mem.cc:113:
  -> Failed to allocate 12884901888 bytes (12884.902 MB) of memory for type CCTK_REAL8.  312 bytes (0.000 MB) are currently allocated in 39 objects
cactus_sim: /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.
Rank 0 with PID 4507 received signal 6
Writing backtrace to 64k2/backtrace.0.txt
Aborted (core dumped)

Note that this has probably been happening for a very long time. The test system just doesn't detect the problem (see #2113).

I'm not 100% sure what this test is supposed to do. It has Carpet::enable_no_storage = yes, so it shouldn't be trying to allocate memory. The test machine has 4 GB of RAM. This could be increased, but we probably don't want to have a test in the ET that requires so much memory.

Keyword:

Comments (8)

  1. Erik Schnetter
    • removed comment

    This test defines a very large grid function that has more that INT_MAX grid points. This can easily happen in production runs. The test ensures that there is no integer overflow in creating such a grid structure.

    For me, this is one of the tests that always succeeds, even if everything else goes wrong, since this test does not produce any output.

  2. Ian Hinder reporter
    • removed comment

    Probably because your machine has enough memory? Can you see why Carpet would abort in the way it does for this test, even though enable_no_storage = yes, if there is not enough memory? Or perhaps, can you reproduce the result on your own machine, by increasing the grid size?

  3. Roland Haas

    enable_no_storage makes Carpet quit just after is has finished executing the WRAGH and PARAMCHECK groups. During those groups memory has already been allocated for reflevel 0 of the grid functions.

    Note that memory for this reflevel is allocated already in SetupGH in the setup_grid_hierarchy function.

    So as long as the OOM is already triggered by reflevel 0 as it is in 64k2.par thenenable_no-storage is not doing anything.

  4. Log in to comment