ValueError in numpy.fromstring

Question

I want to convert these string values to numpy array of int16 datatype

import numpy as np
raw=b''
w="\x01\x02 \x01\x02"
w1="\x01\x03 \x04"
p=w.replace(" ", "")
w1=w1.replace(" ","")
raw +=p
raw +=w1
results = np.fromstring(raw, dtype=np.uint16)
print results

I am getting the error as:

>File "prj1.py", line 11, in <module>
> results = np.fromstring(raw, dtype=np.uint16)
>ValueError: string size must be a multiple of element size

How can I convert these strings to numpy arrray with data type as int16?

What is len(raw)? I'm guessing 7. w1 only has 3 bytes (after removing the space). — hpaulj
– hpaulj, Commented Jul 13, 2017 at 6:06
in 'w1="\x01\x03 \x04"', you should be adding a second character (\x00 before \x04?). If that is not possible, why don't you do it without replacing the spaces and instead do resutl = np.fromstring(raw, dtype=int, sep=' ').astype(uint16)? — Thomas Kühn
– Thomas Kühn, Commented Jul 13, 2017 at 6:06
@hpaulj- Yes the len(raw) is 7. But how can I achieve the array of dtype uint16 ? — Zuhi
– Zuhi, Commented Jul 13, 2017 at 6:35
@joel just to make entirely sure, do you want to get an array with 4 elements? Can you give me the output you expect? I'm not quite sure if the \x04 should be the first or second byte. — Thomas Kühn
– Thomas Kühn, Commented Jul 13, 2017 at 12:25

temporary_user_name · Accepted Answer · 2018-06-10 17:29:28Z

5

As the error message states, if fromstring is fed binary input data, the data length must be a multiple of the element size. This is also stated in the documentation. In your case, the element size is 2, because a uint16 is composed of two bytes. However in your second string, w1, you only provide 1 byte. One way to solve this problem is to add a leading zero to the smaller number:

import numpy as np
raw=b''
w="\x01\x02 \x01\x02"
w1="\x01\x03 \x04"
elements=w.split(' ')+w1.split(' ')
raw=b''.join(['\x00'+e if len(e)==1 else e for e in elements ])
results = np.fromstring(raw, dtype=np.uint16)
print results

This outputs:

[ 513  513  769 1024]

For me this result was surprising. Apparently the bytes are read from left to right (smallest to biggest). I don't know if this is platform specific (I'm on osx) or always like this in numpy. Anyway, if your desired byte order is from right to left, you can reverse the order like so:

raw2=b''.join([e+'\x00' if len(e)==1 else e[1]+e[0] for e in elements])
results2 = np.fromstring(raw2, dtype=np.uint16)
print results2

which results in:

[258 258 259   4]

edited Jun 10, 2018 at 17:29

temporary_user_name

37.4k48 gold badges165 silver badges246 bronze badges

answered Jul 13, 2017 at 12:49

Thomas Kühn

9,9003 gold badges53 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Thomas Kühn Over a year ago

Well, the way you generate the strings, where every number is separated by a space, you do know that. If your input is generated in a different way, the solution may have to be adjusted. If you have only binary input that you do not generate yourself at all, you actually cannot solve this problem, as there is no way to know which numbers have one byte and which ones have two ...

Zuhi Over a year ago

Thankyou :) . But I guess this is useful only when you know the length of the string is 1? But as in the problem I stated here (stackoverflow.com/questions/45042553/… ) the length of the strings are unknown.

Thomas Kühn Over a year ago

@joel -- see my comment in the other question. In my opinion this question is answered.

Siva-Sg · Accepted Answer · 2017-07-13 07:17:37Z

1

From the docs : Raises Value Error : If the string is not the correct size to satisfy the requested dtype and count

You can convert to uint8

raw = ""
w="\x01\x02 \x01\x02"
w1="\x01\x03 \x04"
p=w.replace(" ", "")
w1=w1.replace(" ","")
raw +=p
raw +=w1
results = np.fromstring(raw, dtype=np.uint8)
print(results)

[1 2 1 2 1 3 4]

If you really want to convert to unit16 , you can use astype

results.astype(np.uint16)

array([1, 2, 1, 2, 1, 3, 4], dtype=uint16)

answered Jul 13, 2017 at 7:17

Siva-Sg

2,83121 silver badges29 bronze badges

Collectives™ on Stack Overflow

ValueError in numpy.fromstring

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related