Write pandas DataFrame to HDF in memory buffer

Question

I want to get a dataframe as hdf in memory. The code below results in "AttributeError: '_io.BytesIO' object has no attribute 'put'". I am using python 3.5 and pandas 0.17

import pandas as pd
import numpy as np
import io

df = pd.DataFrame(np.arange(8).reshape(-1, 2), columns=['a', 'b'])
buf = io.BytesIO()
df.to_hdf(buf, 'some_key')

Update: As UpSampler pointed out "path_or_buf" cannot be an io stream (which I find confusing since buf usually can be an io stream, see to_csv). Other than writing to disk and reading it back in, can I get a dataframe as hdf in memory?

Out of curiosity - why would you want to do that?

MaxU - stand with Ukraine
– MaxU - stand with Ukraine

2017-01-06 15:15:26 +00:00
Commented Jan 6, 2017 at 15:15 — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Jan 6, 2017 at 15:15
I just came to the same point, did you manage to solve it?

jlandercy
– jlandercy

2017-09-25 15:14:44 +00:00
Commented Sep 25, 2017 at 15:14 — jlandercy
– jlandercy, Commented Sep 25, 2017 at 15:14

UpSampler · Accepted Answer · 2017-01-06 14:22:02Z

2

Your first argument to df.to_hdf() has to be a "path (string) or HDFStore object" not an io stream. Documentation: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.to_hdf.html

answered Jan 6, 2017 at 14:22

UpSampler

4093 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2133814 Over a year ago

Ahh, didn't see that. to_csv has a similar argument path_or_buf where buf can be an io stream, which is why I got confused.

UpSampler Over a year ago

CSV is a a bulit-in module and HDF is external. I don't know what pandas is using for HDF file access, could be pytables or pyhdf...

Andrew Tapia · Accepted Answer · 2025-08-16 23:58:32Z

0

You can manually manage an in-memory HDFStore to do this. (See the relevant section of the PyTables documentation and this GitHub issue).

import pandas as pd
import numpy as np
import tempfile

df = pd.DataFrame(np.arange(8).reshape(-1, 2), columns=['a', 'b'])
# Although we provide a filename to pd.HDFStore, nothing gets written to disk
# because of the driver_core_backing_store=0 parameter.
with pd.HDFStore(
    tempfile.mktemp(),
    driver="H5FD_CORE",
    driver_core_backing_store=0
) as store:
    store.put("some_key", df, errors="strict", encoding="UTF-8")
    # Store the resulting HDF file bytes to result.
    result = store._handle.get_file_image()

answered Aug 16 at 23:58

Andrew Tapia

1,4832 gold badges12 silver badges23 bronze badges

Comments

Shijo · Accepted Answer · 2017-01-06 14:15:35Z

-1

just try this

df = pd.DataFrame(np.arange(8).reshape(-1, 2), columns=['a', 'b'])
df.to_hdf(path_or_buf='path\to\your\file')

refer pandas.DataFrame.to_hdf

answered Jan 6, 2017 at 14:15

Shijo

9,7813 gold badges23 silver badges31 bronze badges

1 Comment

user2133814 Over a year ago

I want it in memory, not on disk, and would prefer not writing to disk and then reading it back in

Collectives™ on Stack Overflow

Write pandas DataFrame to HDF in memory buffer

3 Answers 3

2 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related