- removed comment
hdf5 deflate fragments memory
Issue #1878
new
In ticket #1874, a problem with memory management of the HDF5 zlib deflate filter was mentioned. This ticket here was opened to collect information about this problem, with the aim of eventually providing a patch to HDF5, submitting a ticket to the HDF5 developers, and possibly implementing that change as patch just for the ET.
The problem is that HDF5 allocates while deflating an output buffer (for the uncompressed data) of the size of the compressed data, and then makes it larger (by a factor of two - another problem), if needed (almost always).
This happens in H5Z_filter_deflate() in H5Zdeflate.c . One way to solve this would be to use the z_strm.total_in value, which holds the size of the uncompressed data, saved by the compressor. However, I am currently unsure whether this is a value we can always rely on. The format specification mentions it as required field, while the zlib documentation talks about it as "may have been saved by the compressor". I didn't look at any actual data.
In any case, if this value is present, we can use it to allocate the output buffer of the correct size. If it isn't present, the best we could do is at least not increase the size by a factor of two, which is known to cause memory fragmentation.
Keyword: HDF5
Comments (2)
-
-
- removed comment
I have used tcmalloc at the same time as the HDF5 decompression, and this was not sufficient to alleviate the problem. It might be that the HDF5 library was not using the tcmalloc malloc (I think I asked you about this in the past Erik; I will have to check the email conversation), or the fragmentation was just too much, so that even tcmalloc was not able to cope. I think that the problem originates in the HDF5 decompression routine, and could be made a lot better by a few small changes there. We should find out whether the size of the original data field can be used or not.
- Log in to comment
- Assignee
- –
- Type
- Priority
- Status
- new
- Component
- Other
- Milestone
- –
- Version
- development version
- Votes
- 0
- Watchers
- 0
I wonder whether memory fragmentation is not a problem caused by the malloc implementation. To test this, you can test with different (newer) versions of libc, or with a malloc replacement. I recommend in particular jemalloc http://www.canonware.com/jemalloc/ -- you build the code as usual, and add
-ljemallocwhen linking. This replaces malloc, free, etc., and the equivalent C++ mechanism. jemalloc claims to be better than GNU malloc is many ways.Another way to avoid fragmentation (that we use in CarpetIOASCII) is to allocate memory not directly via malloc, but rather from a pre-allocated pool. This pool is so large that fragmentation is not an issue on 64-bit systems, since it can be returned to the OS as a whole. Implementing a pool is not too difficult; see Carpet's mempool class (CarpetLib, mem.hh and mem.cc).