TypeError: cannot use a string pattern on a bytes-like object - Py3

Question

I am getting following error with Python3 but the function works fine with Python2

from subprocess import Popen, PIPE, STDOUT
import re

def source_shell():
        
    pattern = re.compile('^(w+)=(.*)$')

    cmd = 'ls /etc/fstab /etc/non-existent-file'
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
    for line in p.stdout:
        line = line.strip()
        if not pattern.match(line):
           print("hurray")
           
source_shell()

Traceback (most recent call last):
  File "main.py", line 15, in <module>
    source_shell()
  File "main.py", line 12, in source_shell
    if not pattern.match(line):
TypeError: cannot use a string pattern on a bytes-like object

What is the safest change to make here, so that it does not break any existing things?

Polyglot answer is appreciated.

Ted Klein Bergman · Accepted Answer · 2020-06-21 18:29:02Z

1

Strings are bytes object in Python 2, while in Python 3 they're string objects. The result you get back from Popen is a bytes object, so you have to convert it to a string (using the decode method) before matching it with your pattern.

from subprocess import Popen, PIPE, STDOUT
import re

def source_shell():
        
    pattern = re.compile('^(w+)=(.*)$')

    cmd = 'ls /etc/fstab /etc/non-existent-file'
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
    for line in p.stdout:
        line = line.strip().decode()  # Decode the bytes-object to a string.
        if not pattern.match(line):
           print("hurray")
           
source_shell()

If you want it to work for both Python 2 and Python 3, I'd suggest converting the pattern to match bytes objects instead of strings.

from subprocess import Popen, PIPE, STDOUT
import re

def source_shell():
        
    pattern = re.compile(b'^(w+)=(.*)$')  # Compile to bytes.

    cmd = 'ls /etc/fstab /etc/non-existent-file'
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
    for line in p.stdout:
        line = line.strip()
        if not pattern.match(line):
           print("hurray")
           
source_shell()

edited Jun 21, 2020 at 18:29

answered Jun 21, 2020 at 18:19

Ted Klein Bergman

9,8075 gold badges35 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

ThinkGeek Over a year ago

Is this polyglot?

Ted Klein Bergman Over a year ago

@LokeshAgrawal Okay, so you're not trying to convert from Python 2 to Python 3, but rather for it to work for both languages?

ThinkGeek Over a year ago

Yes @Ted I am trying to make it work for both the versions

Ted Klein Bergman Over a year ago

@LokeshAgrawal Converting the pattern to a bytes object will work then

Ted Klein Bergman Over a year ago

Actually, they're about the same. The reason I suggested the second option is because decode returns a Unicode string in Python 2 but bytes in Python 3. However, they're compatible for your use case. I think the second version is better though, because you'll always think in bytes instead of different encodings. But both will probably work just fine.

|

Collectives™ on Stack Overflow

TypeError: cannot use a string pattern on a bytes-like object - Py3

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related