how to create hdf5 file from numpy dataset files

Question

I have 1970 .npy files as features for MSVD dataset. I want to create one .hdf5 file from these numpy files.

import os 
import numpy as np
import hdf5


TRAIN_FEATURE_DIR = "MSVD"   

for filename in os.listdir(TRAIN_FEATURE_DIR):
    f = np.load(os.path.join(TRAIN_FEATURE_DIR, filename))
...

Do you know how to load those npy files? How to iterate over the directory? How to create a HDF5 file with h5py? And create groups and datasets? There pieces are all there. — hpaulj
– hpaulj, Commented Dec 13, 2022 at 17:43
@hpaulj I know to load and iterate files only, but I do not know how to create groups and datasets — A_B_Y
– A_B_Y, Commented Dec 13, 2022 at 17:55
Show us how you have applied that knowledge to the current problem. — J_H
– J_H, Commented Dec 13, 2022 at 17:55
The h5py documentation is pretty good. But experiment with simple cases first. — hpaulj
– hpaulj, Commented Dec 13, 2022 at 18:00
We have a pickled object, f. I have no idea what is in the loaded file, nor what attributes f has. You've not cited any h5py documentation, and have not shown any attempts to create an .hdf5 file. I don't know what result you were hoping your question would elicit. Be specific, help us to help you. — J_H
– J_H, Commented Dec 13, 2022 at 18:32

kcw78 · Accepted Answer · 2022-12-13 20:47:02Z

2

Creating a dataset from an array is easy. Example below loops over all .npy files in a folder and creates 1 dataset for each array. (FYI, I prefer glob.iglob() to get the filenames using a wildcard.) Dataset name is the same as the filename.

import glob 
import numpy as np
import h5py

with h5py.File('SO_74788877.h5','w') as h5f:
    for filename in glob.iglob('*.npy'):
        arr = np.load(filename)
        h5f.create_dataset(filename,data=arr)

This code shows how to access the dataset names and values from the H5 file created above. (dataset is a dataset object which behaves like a numpy array in many instances):

with h5py.File('SO_74788877.h5','r') as h5f:
    for name, dataset in h5f.items():
        print(name, dataset.shape, dataset.dtype)

answered Dec 13, 2022 at 20:47

kcw78

8,2184 gold badges18 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to create hdf5 file from numpy dataset files

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related