Unexpected output using subprocess in Python

Question

I am trying to run a shell command from within my Python (version 2.6.5) code, but it is generating different output than the same command run within the shell (bash):

bash:

~> ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1/p' | sed   's/^[ \t]*//;s/[ \t]*$//'
192.168.1.10

Python:

>>> def get_ip():
...     cmd_string = "ifconfig eth0 | sed -rn \'s/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1/p' | sed 's/^[ \t]*//;s/[ \t]*$//\'"
...     process = subprocess.Popen(cmd_string, shell=True, stdout=subprocess.PIPE)
...     out, err = process.communicate()
...     return out
... 
>>> get_ip()
'\x01\n'

My guess is that I need to escape the quotes somehow when running in python, but I am not sure how to go about this.

NOTE: I cannot install additional modules or update python on the machine that this code needs to be run on. It needs to work as-is with Python 2.6.5 and the standard library.

Why are you trying to do the escaping manually instead of just using raw strings so you don't have to? — abarnert
– abarnert, Commented Jun 24, 2013 at 22:11
Why are you using ifconfig (which has unnecessarily hard-to-parse output) rather than ip -o addr list? — Charles Duffy
– Charles Duffy, Commented Jun 24, 2013 at 22:30

abarnert · Accepted Answer · 2013-06-24 22:16:59Z

The reason your code is not working is that you're not escaping enough. You escaped the quotes, but nothing else that needed to be escaped.

Let's look at your intended command line:

ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1/p' | sed   's/^[ \t]*//;s/[ \t]*$//'

And print out your actual command line (just print cmd_string)

ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*//p' | sed 's/^[    ]*//;s/[    ]*$//'

Obviously these aren't the same. The key difference is that your \1 has been replaced with an invisible control character, the one whose ord is 1 (that is, ctrl-A). (You've also replaced each \t with a tab character, but that one probably won't break anything.)

Printing out the repr of the line (print repr(cmd_string)) often helps as well:

"ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\\.){3}[0-9]{1,3}).*/\x01/p' | sed 's/^[ \t]*//;s/[ \t]*$//'"

That \x01 should immediately alert you to what's going on—or, even if you don't understand it, it should alert you to where something is going wrong, so you can do an easier search or write a simpler question at SO.

You should get in the habit of doing both of these whenever you've got something wrong with escaping.

However, usually, the answer is easy: instead of trying to figure out what does and doesn't need to be escaped, just use a raw string:

cmd_string = r"ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1/p' | sed   's/^[ \t]*//;s/[ \t]*$//'"

Now, when you print that out, you get:

ifconfig eth0 | sed -rn 's/inet addr:(([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1/p' | sed   's/^[ \t]*//;s/[ \t]*$//'

Exactly what you wanted.

ChrisProsser · Accepted Answer · 2013-06-24 21:55:31Z

0

In python if you use double quotes around the string then you can use single quotes within that string with no need for escaping and vice versa. However, and backslashes will need to be escaped with an additional \ prefix.

Probably your best bet for debugging this would be to add:

print cmd_string

just after setting cmd_string and then compare this the original version to see if any further characters are missing (these will need escaping too).

answered Jun 24, 2013 at 21:55

ChrisProsser

13.2k6 gold badges38 silver badges45 bronze badges

1 Comment

Paulo Scardine Over a year ago

or you can use "raw" strings (prefix the starting quote with "r"), so you don't need to escape backslashes.

Paulo Scardine · Accepted Answer · 2013-06-24 22:15:39Z

0

Can you afford to install the sh module?

>>> import re, sh
>>> get_ip = lambda: re.search(r'inet addr:(\S+)', 
                               str(sh.ifconfig('eth0'))).group(1)
>>> get_ip()
'10.0.0.202'

A cleaner version:

def get_ip(interface):
    ifconfig = sh.ifconfig(interface)
    match = re.search(r'inet addr:(\S+)', str(ifconfig))
    if match:
        return match.group(1)
    return None

>>> get_ip(eth0)
'192.168.1.10'

Even with subprocess, get rid of sed and use the re module, it will be simpler to read and spare you some troubles with escaping.

edited Jun 24, 2013 at 22:15

answered Jun 24, 2013 at 22:04

Paulo Scardine

78.2k12 gold badges134 silver badges153 bronze badges

Collectives™ on Stack Overflow

Unexpected output using subprocess in Python

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related