Python Struct for Binary Data

Question

I'm trying to read binary files containing a stream of float and int16 values. Those values are stored alternating.

[float][int16][float][int16]... and so on

now I want to read this data file by a python program using the struct functions.

For reading a block of say one such float-int16-pairs I assume the format string would be "fh". Following output makes sense, the total size is 6 bytes

In [73]: struct.calcsize('fh')
Out[73]: 6

Now I'd like to read larger blocks at once to speed up the program...

In [74]: struct.calcsize('fhfh')
Out[74]: 14

Why is this not returning 12?

Generally, what you would want to do is provide some class that buffers your incoming data, and then hands the bytes to you as needed just like the file read function would. I suspect that this is not your problem, though; many operating systems already do this kind of buffering for you. — Robert Harvey
– Robert Harvey, Commented Oct 14, 2014 at 20:16
@RobertHarvey: that class is called file in Python :) e.g., see my answer in the link I've provided. — jfs
– jfs, Commented Oct 14, 2014 at 20:17

Robᵩ · Accepted Answer · 2014-10-14 20:21:05Z

3

Quoting the documentation:

Note By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.

https://docs.python.org/2/library/struct.html

If you want calcsize('fhfh') to be exactly twice calcsize('fh'), then you'll need to specify an alignment character.

Try '<fhfh' or '>fhfh', instead.

answered Oct 14, 2014 at 20:21

Robᵩ

170k20 gold badges251 silver badges323 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PhilippR Over a year ago

Cool, thank you a lot for these very swift responses!

ashwinjv · Accepted Answer · 2014-10-14 20:22:10Z

1

You have to specify the byte order or Endianness as size and alignment are based off of that So if you try this:

>>> struct.calcsize('fhfh')
>>> 14
>>> struct.calcsize('>fhfh')
>>> 12

The reason why is because in struct not specifying an endian defaults to native

for more details check here: https://docs.python.org/3.0/library/struct.html#struct.calcsize

answered Oct 14, 2014 at 20:22

ashwinjv

2,9771 gold badge27 silver badges32 bronze badges

Collectives™ on Stack Overflow

Python Struct for Binary Data

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related