- changed status to open
- removed comment
write index files for checkpoints
Issue #1031
closed
this patch writes index files for checkpoints if output_index is set. This is (only) useful if the checkpoints will be read in by a different number of processes than written. In that case, parsing the index files is faster than parsing the heavy data files.
The two patches provide read and write functionality. The attached C code creates index files from existing HDF5 files (not necessary checkpoints). It is a modified copy of hdf5_extract from the HDF5 thorn.
Keyword: CarpetIOHDF5
Comments (12)
-
reporter
-
- changed status to open
- removed comment
Please apply.
-
reporter
- changed status to resolved
- removed comment
Applied as Carpet hash f44912befe63 "CarpetIOHDF5: teach the checkpointer to write index files" and be246bc0c781 "CarpetIOHDF5: add support for index files to file reader"
-
reporter
- removed comment
I did not commit the index creator stand-alone code. It is functional but a hack. If there is interest or I have nothing else to do (not terribly likely) I'll clean it up and/or write a version based on hdf5_merge rather than hdf5_extract which would make usage simpler. They are fairly trivial modifications of the original tools. Note that the index files created by the tool are not quite identical to Ian's from CarpetIOHDF5 since I write sparse files with the full dataset extents intact whereas CarpetIOHDF5 sets the extents to 1 in each dimension.
-
- removed comment
Sparse files sounds like a better solution. If you have spare time, could you modify CarpetIOHDF5 to also do that? We should also check the VisIt plugin to make sure it works with those.
-
reporter
- changed status to open
- removed comment
One gets sparse files by creating the datasets in the index file with the same extents as in the "heavy" data files but not writing any data into them. The one downside that the sparse files have is that the current hdf5_merge etc. are not smart enough to copy the sparse dataset as a sparse dataset, instead they would write "heavy" file with the datasets all filled with zeroes. Attached please find a patch that does this.
-
reporter
- changed status to open
- removed comment
-
- removed comment
This patch mostly removes code. Is this the correct patch?
-
reporter
- removed comment
Well actually, it did not remove all possible extraneous code :-). One can also get rid of the separate index_dataspace. Having sparse datasets is much simpler than what Ian originally did. I stumbled across them when I did not implement index files quite the way Ian did but simply skipped the H5Dwrite() call in hdf5_create_index.c. Sparse datasets seem to be created if one creates a dataset of a certain size, but then does not actually write data to it (or leaves out holes). So only creating the dataset with the nominal Cactus grid function size but never writing to it makes the file all sparse with all data appearing to have a value of zero when read via H5Dread().
To summarize: The patch removes the separate shape arrays and separate dataspaces for the datasets in index files and creates the index file datasets with the same dataspace as the "heavy" file datasets. It retains the h5shape attribute that was originally introduced for index files even though it is now redundant (since one could call H5Sget_simple_extent_dims on the datasets in the index files).
-
- changed status to open
- removed comment
(One can set the default value for a dataset to make it different from 0.)
Please apply, with an appropriate comment, e.g. the text in comment 9 above.
-
reporter
- changed status to resolved
- removed comment
Applied as hash 437389f2122e "CarpetIOHDF5: make index files sparse files with full datasets" of Carpet.
-
reporter
- edited description
- changed status to closed
- Log in to comment
The create_index tools are a bit of a hack.