Reading and interpreting data from a binary file in Python

Question

I want to read a file byte by byte and check if the last bit of each byte is set:

#!/usr/bin/python

def main():
    fh = open('/tmp/test.txt', 'rb')
    try:
        byte = fh.read(1)
        while byte != "":
            if (int(byte,16) & 0x01) is 0x01:
                print 1
            else:
                print 0
            byte = fh.read(1)
    finally:
        fh.close

    fh.close()

if __name__ == "__main__":
        main()

The error I get is:

Traceback (most recent call last):
  File "./mini_01.py", line 21, in <module>
    main()
  File "./mini_01.py", line 10, in main
    if (int(byte,16) & 0x01) is 0x01:
ValueError: invalid literal for int() with base 16: '\xaf'

Anyone an idea? I didn't succeed using the struct and the binascii modules.

Read stackoverflow.com/questions/306313/… before you get this working and open a dupe of that question ;) — user395760
– user395760, Commented Oct 15, 2010 at 14:31

Scott Griffiths · Accepted Answer · 2010-10-15 18:51:26Z

61

Try using the bytearray type (Python 2.6 and later), it's much better suited to dealing with byte data. Your try block would be just:

ba = bytearray(fh.read())
for byte in ba:
    print byte & 1

or to create a list of results:

low_bit_list = [byte & 1 for byte in bytearray(fh.read())]

This works because when you index a bytearray you just get back an integer (0-255), whereas if you just read a byte from the file you get back a single character string and so need to use ord to convert it to an integer.

If your file is too big to comfortably hold in memory (though I'm guessing it isn't) then an mmap could be used to create the bytearray from a buffer:

import mmap
m = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
ba = bytearray(m)

edited Oct 15, 2010 at 18:51

answered Oct 15, 2010 at 14:40

Scott Griffiths

22k8 gold badges58 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

brichins Over a year ago

+1 for the large file solution, I was saved from much hair-tearing. You sir are a gentleman and a scholar.

nmichaels · Accepted Answer · 2010-10-15 15:15:23Z

9

You want to use ord instead of int:

if (ord(byte) & 0x01) == 0x01:

edited Oct 15, 2010 at 15:15

answered Oct 15, 2010 at 14:27

nmichaels

51.2k12 gold badges113 silver badges137 bronze badges

2 Comments

Scott Griffiths Over a year ago

and you really should compare integers with ==, not by their identities!

nmichaels Over a year ago

Good point. I thought about it, but didn't change it when I pasted that...fixed.

tzot · Accepted Answer · 2010-10-15 19:22:23Z

5

One way:

import array

filebytes= array.array('B')
filebytes.fromfile(open("/tmp/test.txt", "rb"))
if all(i & 1 for i in filebytes):
    # all file bytes are odd

Another way:

fobj= open("/tmp/test.txt", "rb")

try:
    import functools
except ImportError:
    bytereader= lambda: fobj.read(1)
else:
    bytereader= functools.partial(fobj.read, 1)

if all(ord(byte) & 1 for byte in iter(bytereader, '')):
    # all bytes are odd
fobj.close()

answered Oct 15, 2010 at 19:22

tzot

96.6k30 gold badges151 silver badges210 bronze badges

Collectives™ on Stack Overflow

Reading and interpreting data from a binary file in Python

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related