Parsing binary data into separate variables in Python?

Question

Say I open a file in Python that contains a series of binary data.

with open(sys.argv[1]) as data_file:
    logData = data_file.read()

I basically want to create a loop saying:

for each_word in logData:
       var1 = first 8 bytes
       var2 = next 16 bytes
       var3 = next 8 bytes

C code to generate my binary file:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
        FILE *file;
        int  buffer = 0x03000201;
        int  buffer2= 0x010203;
        file = fopen("test.bin", "wb");

        if (file != NULL)
        {
                fwrite(&buffer, sizeof(buffer), 1, file);
                fwrite(&buffer2, sizeof(buffer2), 1, file);
                fclose(file);
        }

        return 0;
}

and this basically continues until the loop is over, iterating through the bytes of data. How is this possible?

ankostis · Accepted Answer · 2021-04-04 07:40:13Z

8

Use the struct module, it will also allow you to interpret the binary data in many ways; you need to define the types in a string format documented with that library:

struct.unpack('=HHf255s', bytes)

The above example expects native byte-order, two unsigned shorts, a float and a string of 255 characters.

Your code example becomes:

for each_word in logData:
    var1, var2, var3 = struct.unpack('8s16s8s', each_word)

In case you get an error TypeError: 'str' does not support the buffer interface, that is because bytes are expected, but you passed a str, so convert it to bytes and specify the encoding (in this example, UTF-8):

for each_word in logData:
    var1, var2, var3 = struct.unpack('8s16s8s', bytes(each_word, 'utf-8'))

But maybe the case that your 8/16 byte strings are long integers? In that case use the appropriate format for struct.

EDIT: it turns out you wanted to read 8 bits (not bytes), then the next 16 bits, then the next 8 bits, so you can read it as one (unsigned?) byte, one short, and another byte. The format string you should use is '=bhb' (or '=BHB' for unsigned). Example:

import struct
with open('test.bin','rb') as f:
    var1, var2, var3 = struct.unpack('=BHB', f.read(4))
    print(var1, var2, var3)

edited Apr 4, 2021 at 7:40

ankostis

9,6703 gold badges53 silver badges68 bronze badges

answered Jun 3, 2015 at 20:43

fferri

19k8 gold badges57 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Taryn Over a year ago

Comments are not for extended discussion; this conversation has been moved to chat.

Collectives™ on Stack Overflow

Parsing binary data into separate variables in Python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related