speed up reading metadata of HDF5 files in visit

Comments (10)

1. 

   Roland Haas reporter

   • changed status to open
   • removed comment

   • 2013-11-19T23:38:28+00:00
2. 

   Ian Hinder

   • removed comment

   Is this beneficial even when using index files? We do a similar thing in SimulationTools. Take a look at https://bitbucket.org/simulationtools/simulationtools/src/8e0d40844bc9831bb3ba80e9fd453371c0574327/CarpetIOHDF5.m?at=master around line 332 "Low-level interface for determining what's available in a file". Essentially we read in all the dataset names and parse them, and build up a lookup table which is then used to find datasets by the key/value pairs from the name. Note that (as explained in the comment in the file) we use the terminology "attribute" here to refer to the X=Y parts of the dataset name, not the HDF5 attribute (this should probably be changed). The HDF5 attributes are read using the Annotations function, but this is only called when reading the data (to get the origin and delta) or something (like cctk_time) that is only available in HDF5 attributes.

   • 2013-11-20T05:58:15+00:00
3. 

   Roland Haas reporter

   • removed comment

   I made up some test using a file with very many (~400k) datasets (9.2GB which does not quite fit into main memory). I give times to iterate over the file and read the metadata (using gettimeofday):

   With patch and without index files: 26s With patch and with index files: 18s Without patch and without index files: 27s Without patch and with index files: 19s

   Doing the same with a large 3d dataset (~16GB) I find:

   With patch and without index files: 12s With patch and with index files: 10s Without patch and without index files: 12s Without patch and with index files: 10s

   So ti seems as if this has not measurable impact in speed (index files however help). This would mean that there is no need (anymore?) to try and parse information out of the dataset name and one can just query the attributes.

   • 2013-11-20T21:05:35+00:00
4. 

   Erik Schnetter

   • removed comment

   Are index files widely used?

   • 2014-02-11T21:30:54+00:00
5. 

   Roland Haas reporter

   • removed comment

   Not as far as I can tell. I usually don't use them (since I forget to turn them on).

   • 2014-02-12T16:38:25+00:00
6. 

   Ian Hinder

   • removed comment

   I think this ticket can be closed, as the enhancement, once benchmarked, showed no speed improvement. Roland, do you agree? Or do you want to do more testing?

   • 2014-02-13T12:42:20+00:00
7. 

   Roland Haas reporter

   • removed comment

   Ian: I'll likely just apply it since it cannot hurt and the same logic already exists for the other attributes. I have more CarpetHDF5 changes anyway so I can just bunch them all up and try to get them into VisIt in one go. I am still confused that apparently in the Mathematica reader using string parsing was found to be a good thing but that the Carpet HDF5 reader sees so little benefit. Possibly the issue is that I used postprocessed files that might be "nicer" than what the simulation would produce directly.

   • 2014-02-13T12:49:28+00:00
8. 

   Ian Hinder

   • changed status to open
   • removed comment

   Please apply if still considered appropriate.

   • 2014-08-26T05:07:08+00:00
9. 

   Roland Haas reporter

   • changed status to resolved
   • removed comment

   Applied as rev 57 of CarpetHDF5.

   • 2014-09-15T07:34:13+00:00
10. 

    Roland Haas reporter

    • edited description
    • changed status to closed

    • 2019-02-21T20:12:02+00:00
11. Log in to comment