IO corruption on SDSC oasis file systems

I am experiencing file corruption in ASCII output produced by the Cactus code on Comet's (and Gordon's as far as I remember) scratch file systems. This manifests as lines of output being mashed together in the output file.

I have added strace calls to my job script to capture all arguments to the OS's write() function and re-created the write() calls based on this. Those write calls, when replayed on a login node, do produce a correct (no mashed lines) file.

All output to the file in question was from rank 0 only even though the code used MPI and ran on two MPI ranks.

The same code and number of MPI ranks produces a correct output file when run on the $HOME file system.

Thus it seems to me as if there may be an issue with the file system. I can try and reduce the test case to a more minimal example (right now it is a full simulation even though it runs only for <1minute) .

You can find the job script (for account, SLURM options etc) here:

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/SubmitScript

the script that launches the MPI executable here:

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/RunScript

the strace output here:

/home/rhaas/strace/strace.1882[67].log

and the awk script to recreate the write calls is:

gawk -vFS='"' '/write.*\/grid-coordinates.xy.asc/{print "printf \""$2"\""}' ~/strace.18826.log >recreate.sh

The corrupted line is eg. line 161 of

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/TEST/sim/CarpetIOASCII/newsep/grid-coordinates.xy.asc

which reads

1 4 3 4 1 0.1666666666660.505076272276105285714 etc

but should read

1 4 3 4 1 0.166666666666667 -0.0714285714285714

I can avoid the file corruption by flushing the output file after each line.

I am wondering if there is anything known about this or if there is a workaround that does not boil to first writing all data to a file system local to the compute node and copying to /oasis/scratch after the job is finished (how much local space would be available since I would also have to do so for eg checkpoint files and 3d hdf5 output).

Keyword: Comet
Keyword: Gordon
Keyword: SDSC

Comments (5)