Python/Numpy: Efficiently store non-sparse large symmetric arrays?

Question

I am trying to store data on my hard drive that comes in the form of 2 million symmetric 100x100 matrices. Almost all elements of these matrices are non-zero. I am currently saving this data in 200 npy files; each of which has size 5.1GB and contains 100000x100x100 numpy array. This takes up more than 1TB of hard drive space.

Is there anyway that I can use the fact that the matrices are symmetric to save space on my hard drive?

You can write your own class which only stores one triangular part (+ diag) in a flattened form (while storing the flattening-order Fortran vs. C (maybe not needed; can't wrap my head around currently); part of numpy-array). But if you are using compression (savez_compressed) during saving, i expect it to be nearly as efficient (guessing). At least for smaller arrays as compression is probably typical LZW-like using some window-size. — sascha
– sascha, Commented Jan 19, 2018 at 15:47
I would think so (the API is somewhat unintuitive). But as i said: easy to check! — sascha
– sascha, Commented Jan 19, 2018 at 15:57
OK, I'll check the documentation for savez_compressed and give it a try. Thanks! — Alice Schwarze
– Alice Schwarze, Commented Jan 19, 2018 at 15:58
scipy distance pdist calculations return a condensed form, and can switch between that and squareform. Condensed is the triangular values as 1d. — hpaulj
– hpaulj, Commented Jan 19, 2018 at 16:33
@AliceSchwarze, great, good to know. i might play around with savez_compressed next time. — jpp
– jpp, Commented Jan 29, 2018 at 1:52

user545424 · Accepted Answer · 2018-01-19 23:59:01Z

2

To store only the upper half of the matrix (including the diagonal) you should be able to do something like:

import numpy as np

data = np.load([filename])

flat = []
for a in data:
    flat.append(a[np.triu_indices(100)])

np.savez([filename], *flat)

And then to load them back:

import numpy as np

flat = np.load([filename])

data = []

for name, a in flat:
    arr = np.zeros((100,100),dtype=[dtype])
    arr[np.triu_indices(100)] = a
    arr = arr + arr.T - np.diag(arr.diagonal)
    data.append(arr)

data = np.array(data)

answered Jan 19, 2018 at 23:59

user545424

16.3k11 gold badges61 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jpp · Accepted Answer · 2018-01-20 20:20:09Z

Consider HDF5. There are many options for compression via h5py python library. Two of the most popular are lzf (fast decompression, moderate compression ratio) and gzip (slower decompression, good compression ratio). With gzip, you can choose compression level. In addition, gzip and lzf allow shuffle filters to improve compression ratios.

For dense arrays of uncompressed size ~8GB (csv), I typically see 75% reduction after applying lzf in HDF5. I don't expect so large a benefit from npy to HDF5, but it could still be significant.

Another benefit is HDF5 supports lazy reading. In python you can do this directly through h5py or via dask.array.

If you wish to go down this route, h5py documentation has sample high-level code.

Disclaimer: I have no affiliation with h5py or dask. I just find these libraries useful for high-level data analysis.

Collectives™ on Stack Overflow

Python/Numpy: Efficiently store non-sparse large symmetric arrays?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related