Huge HDF5 file in Matlab and importing it in Python

Question

I have created a hdf5 file in Matlab with a Matrix size of (1 x 19,000,000). The file had a size of 150 megabytes.

My question is on how to find the perfect chunk size and deflate ratio? After playing around I have discovered that a chunk size of 1 x 1,000,000 with deflate set to 7 achieves a file of 100 megabytes.
My second problem is that I am unable to import this file in Python

Matlab

h5create('Xn.h5','/rawdata',size (data),'ChunkSize',[1 1000000],'Deflate',7 )

Python

import h5py
filename = 'Xn.h5'
f = h5py.File(filename, 'r')

print("Keys: %s" % f.keys())

I expected that Python will handle the data smoothly just as matlab but this never happened

A hdf5 file of 150 mb is certainly not huge. You should not need to worry about compression at all in this regime. Could you post the error message you get when attempting to read it in python. — Florian Drawitsch
– Florian Drawitsch, Commented Feb 11, 2019 at 10:09
@FlorianDrawitsch, thanks for your comment. I am not getting an error but I am also not able to read the the dataset inside my hdf file. My plan is to use the data and plot it and python os running in the backhround without error or data — Youssef Yassine
– Youssef Yassine, Commented Feb 11, 2019 at 10:31
If you are not getting an error, what makes you think the dataset is not read properly then? What is returned for e.g. f[list(f.keys())[0]] — Florian Drawitsch
– Florian Drawitsch, Commented Feb 11, 2019 at 11:54
@FlorianDrawitsch, executing a_group_key = list(f.keys())[0] ;data1 = list (f1[a_group_key]) takes 2 hours — Youssef Yassine
– Youssef Yassine, Commented Feb 11, 2019 at 11:59
Please also note that the data1 = list (f1[a_group_key]) command you issue converts the returned data into a list. Please execute exactly what I suggested: f[list(f.keys())[0]] — Florian Drawitsch
– Florian Drawitsch, Commented Feb 11, 2019 at 12:18

Florian Drawitsch · Accepted Answer · 2019-02-13 10:27:21Z

0

Ok as it seems to turn out, this question is more related to "How do I access my data in a hdf5 container in python?".

You find a very good quick start guide here.

The process of accessing your data works like this:

import h5py
f = h5py.File(filename, 'r') 
key = list(f.keys())[0]
dataset = f[key]

# To retrieve e.g. the first 10 elements of a 1D dataset execute
data = dataset[0:9]

answered Feb 13, 2019 at 10:27

Florian Drawitsch

7055 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Huge HDF5 file in Matlab and importing it in Python

Matlab

Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Matlab

Python

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related