How to append data to h5 file in python?

Question

I would like to add more and more data to the HDF5 file as the data comes in. I tried the following: First create a dataset with first array and then try to add one more value to the h5 file by resizing it.

import os
import h5py
import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 9, 8, 87, 2, 3, 5, 12, 14, 16]).astype(int)
y = 9
path = "out.h5"
with h5py.File(path, "a") as f:
    dset = f.create_dataset("somedata", data=x, maxshape=(None,))
    dset[:] = y
    print(dset.shape)

    for i in range(3):
        dset.resize(dset.shape[0] + 1, axis=0)
        dset["somedata"] = y

some = h5py.File("out.h5", "r")
for x in some["somedata"]:
    print(x)

But it is throwing me an error:

File "/anaconda3/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 582, in __setitem__
    raise TypeError("Illegal slicing argument (not a compound dataset)")
TypeError: Illegal slicing argument (not a compound dataset)

Why are you using dset["somedata"]? dset is the dataset object. You used it correctly with dset[:]=y. I think some["somedata"] is correct. — hpaulj
– hpaulj, Commented Jul 27, 2020 at 4:17
I'd imagine the easiest way is using pandas's df.to_hdf with mode='a': pandas.pydata.org/pandas-docs/stable/reference/api/… — BallpointBen
– BallpointBen, Commented Jul 27, 2020 at 4:21

Krishnakanth Allika · Accepted Answer · 2020-07-27 04:52:54Z

1

It appears you are trying to resize the dataset and overwrite it with y. Can you try this?

import h5py
import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 9, 8, 87, 2, 3, 5, 12, 14, 16]).astype(int)
y = 9
path = "out.h5"
with h5py.File(path, "a") as f:
    dset = f.create_dataset('somedata', data=x, maxshape=(None,))
    dset[:] = y
    print(dset.shape)

    for i in range(3):
        dset.resize(dset.shape[0] + 1, axis=0)
        dset[:] = y

with h5py.File("out.h5", "r") as some:
    for x in some["somedata"]:
        print(x)

answered Jul 27, 2020 at 4:52

Krishnakanth Allika

8106 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to append data to h5 file in python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related