File reading from PHP using python script

Question

Okay, this is driving me crazy. I have a small file. Here is the dropbox link https://www.dropbox.com/s/74nde57f07jj0zj/transcript.txt?dl=0.

If I try to read the content of the file using python f.read(), I can easily read it. But, if I try to run the same python program using php shell_exec(), the file read fails. This is the error I get.

Traceback (most recent call last): 
  File "/var/www/python_code.py", line 2, in <module>
    transcript = f.read() 
  File "/opt/anaconda/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 107: ordinal not in range(128)

I have checked all the permission issues and there is no problem with that.

Can anyone kindly shed some light?

Here is my python code.

f = open('./transcript/transcript.txt', 'r')
transcript = f.read()
print(transcript)

Here is my PHP code.

$output = shell_exec("/opt/anaconda/bin/python /var/www/python_code.py");

Thank you!

EDIT: I think the problem is in the file content. If I replace the content with simple 'I eat rice', then I can read the content from php. But the current content cannot be read. Still don't know why.

Does python_code.py have read access for others? Try chmod o+r python_code.py then? Just tried on my server. — user2066448
– user2066448, Commented May 24, 2015 at 2:49
Is it because the script is being executed from a different directory, thus making a relative url incorrect? — Paul Rooney
– Paul Rooney, Commented May 24, 2015 at 2:52
The URL is correct and the python_code has access by others. — Naffi
– Naffi, Commented May 24, 2015 at 2:58
I am trying to say that your php shell might not be executing the python script from the directory it is actually in. If this were the case './transcript/transcript.txt' a relative url might not be valid from the directory your script is executing from. You can try printing os.getcwd() (needs import os) to see which directory the script is executed from. Is this what you expected? Another thing to try is to use an absolute url in your python script /var/www/transcript/transcript.txt. It's hard to tell the exact error without your actual error message. — Paul Rooney
– Paul Rooney, Commented May 24, 2015 at 3:12

abarnert · Accepted Answer · 2015-05-24 21:57:17Z

1

The problem appears is that your file contains non-ASCII characters, but you're trying to read it as ASCII text.

Either it is text, but is in some encoding or other that you haven't told us (probably UTF-8, Latin-1, or cp1252, but there are countless other possibilities), or it's not text at all, but rather arbitrary binary data.

When you open a text file without specifying an encoding, Python has to guess. When you're running from inside the terminal or whatever IDE you use, presumably, it's guessing the same encoding that you used in creating the file, and you're getting lucky. But when you're running from PHP, Python doesn't have as much information, so it's just guessing ASCII, which means it fails to read the file because the file has bytes that aren't valid as ASCII.

If you want to understand how Python guesses, see the docs for open, but briefly: it calls locale.getpreferredencoding(), which, at least on non-Windows platforms, reads it from the locale settings in the environment. On a typical linux system that's not new enough to be based on systemd but not too old, the user's shell will be set up for a UTF-8 locale, but services will be set up for C locale. If all of that makes sense to you, you may see a way to work around your problem. If it all sounds like gobbledegook, just ignore it.

If the file is meant to be text, then the right solution is to just pass the encoding to the open call. For example, if the file is UTF-8, do this:

f = open('./transcript/transcript.txt', 'r', encoding='utf-8')

Then Python doesn't have to guess.

If, on the other hand, the file is arbitrary binary data, then don't open it in text mode:

f = open('./transcript/transcript.txt', 'rb')

In this case, of course, you'll get bytes instead of str every time you read from it, and print is just going to print something ugly like b'aq\x9bz' that makes no sense; you'll have to figure out what you actually want to do with the bytes instead of printing them as a bytes.

edited May 24, 2015 at 21:57

answered May 24, 2015 at 21:48

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Naffi Over a year ago

The file content is meant to be text. So, I have taken the first suggestion and the error is gone. Thanks a lot! Now, I find another error. When I am printing the content of the file, I get "UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 107: ordinal not in range(128)" error. I suppose something need to be specified in the print() command as well?

ls129 Over a year ago

turns out on ubuntu by default open() assumes binary file and on anaconda it assumes text. I ran the same code serving a web page on both PC and linux, using flask. on the windows/pc, favicon was not showing up and fontawsome was showing empty squares. took a while but it turns out adding the 'rb' parameter to open() solved the problems. thanks!

Collectives™ on Stack Overflow

File reading from PHP using python script

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related