- removed comment
Carpet/Carpet/test/64k2.par test fails on Jenkins test machine
The test Carpet/Carpet/test/64k2.par exits with nonzero and reports:
INFO (Carpet): There are 0 3-dimensional grid arrays in 0 groups INFO (Carpet): (The number of variables counts all time levels) ERROR from host 7ce14e5707a0 process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn CarpetLib, file /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/CarpetLib/mem.cc:113: -> Failed to allocate 12884901888 bytes (12884.902 MB) of memory for type CCTK_REAL8. 312 bytes (0.000 MB) are currently allocated in 39 objects ERROR from host 7ce14e5707a0 process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn CarpetLib, file /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/CarpetLib/mem.cc:113: -> Failed to allocate 12884901888 bytes (12884.902 MB) of memory for type CCTK_REAL8. 312 bytes (0.000 MB) are currently allocated in 39 objects cactus_sim: /home/jenkins/workspace/EinsteinToolkitGeneric/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank 0 with PID 4507 received signal 6 Writing backtrace to 64k2/backtrace.0.txt Aborted (core dumped)
Note that this has probably been happening for a very long time. The test system just doesn't detect the problem (see #2113).
I'm not 100% sure what this test is supposed to do. It has Carpet::enable_no_storage = yes, so it shouldn't be trying to allocate memory. The test machine has 4 GB of RAM. This could be increased, but we probably don't want to have a test in the ET that requires so much memory.
Keyword:
Comments (8)
-
-
reporter - removed comment
Probably because your machine has enough memory? Can you see why Carpet would abort in the way it does for this test, even though enable_no_storage = yes, if there is not enough memory? Or perhaps, can you reproduce the result on your own machine, by increasing the grid size?
-
enable_no_storage
makes Carpet quit just after is has finished executing the WRAGH and PARAMCHECK groups. During those groups memory has already been allocated for reflevel 0 of the grid functions.Note that memory for this reflevel is allocated already in
SetupGH
in thesetup_grid_hierarchy
function.So as long as the OOM is already triggered by reflevel 0 as it is in 64k2.par then
enable_no-storage
is not doing anything. -
Pull request is here: https://bitbucket.org/eschnett/carpet/pull-requests/30/carpet-disable-allocation-storage-for-gf/diff
The change explicitly disables memory allocation for CCTK_GF groups (disabling memory for all grid variables leads to a segfault) in Carpet’s storage code by allocation 0 timelevels for them.
-
@Erik Schnetter @Ian Hinder Please review.
-
- edited description
- changed status to open
-
Unless objected I will push this change after 2019-12-16
-
- changed status to resolved
- Log in to comment
This test defines a very large grid function that has more that INT_MAX grid points. This can easily happen in production runs. The test ensures that there is no integer overflow in creating such a grid structure.
For me, this is one of the tests that always succeeds, even if everything else goes wrong, since this test does not produce any output.