UnicodeDecodeError message in python

Question

I am relatively new to Python programming. I am using Python 3.3.2 on Windows XP.

My program was working and then all of a sudden I got a UnicodeDecodeError error message.

The exec.py file looks like this:

import re
import os,shutil  
f=open("C:/Documents and Settings/hp/Desktop/my_python_files/AU20-10297-2_yield_69p4_11fails_2_10_14python/a1.txt","a")
for r,d,fi in os.walk("C:/Documents and Settings/hp/Desktop/my_python_files/AU20-10297-2_yield_69p4_11fails_2_10_14python"):
for files in fi:
    if files.startswith("data"):
        g=open(os.path.join(r,files))
        shutil.copyfileobj(g,f)
        g.close()
f.close()

keywords = ['FAIL']
pattern = re.compile('|'.join(keywords))



inFile = open("a1.txt")
outFile =open("failure_info", "w")
keepCurrentSet = False
for line in inFile:
if line.startswith("                                 Test Results"):
    keepCurrentSet = False

if keepCurrentSet:
    outFile.write(line)

if line.startswith("Station ID "):
    keepCurrentSet = True

#if 'FAIL' in line in inFile:
   # outFile.write(line)

if pattern.search(line):
    outFile.write(line)

inFile.close()
outFile.close()

Now, a1.txt is initially an empty seed text file used for collecting data from the data files. I got the following error messages:

Traceback (most recent call last):
  File "C:\Documents and Settings\hp\Desktop\my_python_files\AU20-10297-2_yield_69p4_11fails_2_10_14python\exec.py", line 8, in <module>
    shutil.copyfileobj(g,f)
  File "C:\Python33\lib\shutil.py", line 68, in copyfileobj
    buf = fsrc.read(length)
  File "C:\Python33\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 754: character maps to <undefined>

Can anyone help me fix the code so it is more robust?

Martijn Pieters · Accepted Answer · 2014-02-15 19:15:41Z

2

You have opened the files in text mode, which means Python will try and decode the contents to Unicode. You'd normally need to specify the correct codec for the file (or Python will use your platform default), but are just copying files across with shutil.copyfileobj() here, and decoding is not needed.

Open the files in binary mode instead.

f = open(..., 'ab')
for r,d,fi in os.walk(...):   
    for files in fi:  
        if files.startswith("data"):
            g = open(os.path.join(r, files), 'rb')
            shutil.copyfileobj(g,f)

Note the addition of the b to the filemode.

You probably want to use the file objects as context managers so they are closed for you, automatically:

with open(..., 'ab') as outfh:
    for r,d,fi in os.walk(...):   
        for files in fi:  
            if files.startswith("data"):
                with open(os.path.join(r, files), 'rb') as infh:
                    shutil.copyfileobj(infh, outfh)

answered Feb 15, 2014 at 19:15

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user3313975 Over a year ago

Thanks for solving my problem!

Collectives™ on Stack Overflow

UnicodeDecodeError message in python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related