How to keep hdf5 binary of a pandas dataframe in-memory?

Question

I would like to get the byte contents of a pandas dataframe exported as hdf5, ideally without actually saving the file (i.e., in-memory).

On python>=3.6, < 3.9 (and pandas==1.2.4, pytables==3.6.1) the following used to work:

import pandas as pd
with pd.HDFStore(
    "in-memory-save-file",
    mode="w",
    driver="H5FD_CORE",
    driver_core_backing_store=0,
) as store:
    store.put("my_key", df, format="table")
    binary_data = store._handle.get_file_image()

Where df is the dataframe to be converted to hdf5, and the last line calls this pytables function.

However, starting with python 3.9, I get the following error when using the snippet above:

File "tables/hdf5extension.pyx", line 523, in tables.hdf5extension.File.get_file_image
tables.exceptions.HDF5ExtError: Unable to retrieve the size of the buffer for the file image.  Plese note that not all drivers provide support for image files.

The error is raised by the same pytables function linked above, apparently due to issues while retrieving the size of the buffer for the file image. I don't understand the ultimate reason for it, though.

I have tried other alternatives such as saving to a BytesIO file-object, so far unsuccessfully.

How can I keep the hdf5 binary of a pandas dataframe in-memory on python 3.9?

This is a related (albeit old) discussion on github where the call to get_file_image() is suggested — swimmer
– swimmer, Commented May 12, 2021 at 9:13

swimmer · Accepted Answer · 2021-05-18 12:59:44Z

1

The fix was to do conda install -c conda-forge pytables instead of pip install pytables. I still don't understand the ultimate reason behind the error, though.

answered May 18, 2021 at 12:59

swimmer

3,5423 gold badges35 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to keep hdf5 binary of a pandas dataframe in-memory?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related