subprocess python 3 check_output not same as shell command?

Question

I am trying to use the subprocess module in python but its a bit tricky to get working. Here's my code

import sys
import os
import subprocess
import shlex

def install_module(dir_path, command):
    c = shlex.split(command)
    os.chdir(dir_path)
    try:
        p = subprocess.check_output(c, shell=True)
    except subprocess.CalledProcessError as e:
        #print('install failed for: ' + dir_path + ' ' + command)
        print(e.output)

def main():
    install_module('D:\installed_software\python modules\kennethreitz-requests-e95e173'
                   , 'python setup.py install')
    install_module('D:\installed_software\python modules\psycopg2-2.6.1'
                   , 'python setup.py build')
    install_module('D:\installed_software\python modules\psycopg2-2.6.1'
                   , 'python setup.py install')
    install_module('D:\installed_software\python modules\pypyodbc-1.3.3\pypyodbc-1.3.3'
                   , 'python setup.py install')

if __name__ == "__main__":
    sys.exit(main())

and my output:

install failed for: D:\installed_software\python modules\psycopg2-2.6.1 python setup.py build
b'running build\r\nrunning build_py\r\nrunning build_ext\r\n'
install failed for: D:\installed_software\python modules\psycopg2-2.6.1 python setup.py install
b'running install\r\nrunning build\r\nrunning build_py\r\nrunning build_ext\r\n'

but when i try running this command normally through cmd i get the below output

D:\installed_software\python modules\psycopg2-2.6.1>python setup.py build
running build
running build_py
running build_ext
Error: pg_config executable not found.

Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:

    python setup.py build_ext --pg-config /path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'.

why are they different. I have played with this module a little bit and its really hard to get it to put input back in and to read output from its current shell. Any help would be greatly appreciated

UPDATE:

So the below code works yay! thanks J.F! But I am still having issues with

sys.sterr.flush()

my code with the sys.sterr.flush() line commented

import sys
import os
from subprocess import CalledProcessError, STDOUT, check_output
import shlex

import sys
import os
from subprocess import CalledProcessError, STDOUT, check_output
import shlex

def run_in_path(command, dir_path):
    #c = shlex.split(command)
    #os.chdir(dir_path)
    try:
        p = check_output(command, cwd=dir_path, stderr=STDOUT)
    except CalledProcessError as e:
        sys.stderr.write(e.output.decode("utf-8"))
        #sys.sterr.flush()
        return e.returncode
    else:
        return 0

def main():
    run_in_path('python setup.py build',
                'D:\installed_software\python modules\kennethreitz-requests-e95e173')
    run_in_path('python setup.py build',
                   'D:\installed_software\python modules\psycopg2-2.6.1')
    run_in_path('python setup.py install',
                   'D:\installed_software\python modules\psycopg2-2.6.1')
    run_in_path('python setup.py install',
                   'D:\installed_software\python modules\pypyodbc-1.3.3\pypyodbc-1.3.3')

if __name__ == "__main__":
    sys.exit(main())

The error I get when i run sys.sterr.flush() is

    sys.sterr.flush()
AttributeError: 'module' object has no attribute 'sterr'

unrelated: don't use python setup.py install, prefer pip install -r requirements.txt instead: it may enable proper uninstall and it could speedup reinstallations (due to wheel support). — jfs
– jfs, Commented Jun 22, 2015 at 12:24
Yeah the only reason im using setup install is becasue the server i am on does not have access to the internet. So i have to have all the install files locally — Justin S
– Justin S, Commented Jun 22, 2015 at 23:31
Hey J.F Yeah im running python 3.3 so it doesnt have pip installed by default. Also when i try download pip using python get-pip.py I have no internet access so I cannot download it. Reading on the internet I could just get the .wheel of pip to install but it is not supported or recommended. For now I am happy just to run through the installs using python but thanks for the feedback will keep it in mind if this project size starts to radically increase — Justin S
– Justin S, Commented Jun 22, 2015 at 23:53
AttributeError is due to the typo: compare sys.stderr vs. sys.sterr — jfs
– jfs, Commented Jun 23, 2015 at 0:09

jfs · Accepted Answer · 2015-06-22 12:22:18Z

1

shlex.split() syntax is different from the one used by cmd.exe (%COMSPEC%)
use raw-string literals for Windows paths i.e., use r'c:\Users' instead of 'c:\Users'
you don't need shell=True here and you shouldn't use it with a list argument
you don't need to split the command on Windows: string is the native interface

You could use cwd parameter, to run the command in the specified directory:

#!/usr/bin/env python3
import sys
from subprocess import CalledProcessError, STDOUT, check_output

def run_in_path(command, dir_path):
    try: #NOTE: show output only if an error happens   
        ignored = check_output(command, cwd=dir_path, stderr=STDOUT) 
    except CalledProcessError as e:
        sys.stderr.buffer.write(e.output) 
        sys.stderr.buffer.flush()
        return e.returncode
    else:
        return 0

edited Jun 22, 2015 at 12:22

answered Jun 22, 2015 at 12:17

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Justin S Over a year ago

Thanks for your reply J.F. This module is starting to make a lot more sense. I am having issues with the line sys.stderr.buffer.write(e.output) i get this error sys.stderr.buffer.write(e.output) AttributeError: 'PseudoOutputFile' object has no attribute 'buffer'. I have no idea whats going on

jfs Over a year ago

@JustinS: use regular Python to run your script.

Justin S Over a year ago

Hey J.F not to sure what you mean by this? I run this script by hitting 'F5' in IDLE?

jfs Over a year ago

IDLE replaces sys.stderr. Run the script from the command-line instead e.g., py your_script.py

Collectives™ on Stack Overflow

subprocess python 3 check_output not same as shell command?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related