0

I am writing a Python program that needs to return the active hosts scanned in one of my vulnerability scans. I have used this method before returning XML, but when I try to tack on these extra programs such as cut and grep I run into issues. Perhaps it doesn't like "pipes" | or maybe I am doing something completely wrong here with my commas but I have tried all sorts of things and cant seems to get it to return the result like it does when I run the command standalone from the command line. Thanks very much for any help that is provided.

def activeHostsQuery():
    args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']

    activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0]
    return activeHostsNumber
Improve this question
5
  • Have you tried adding the "shell=True" argument to subprocess.Popen? Commented May 25, 2012 at 1:19
  • activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE, shell=TRUE).communicate()[0] ..... this gives me a "global name "TRUE" not defined error Commented May 25, 2012 at 1:23
  • It needs to be 'True', not 'TRUE' - case matters. Commented May 25, 2012 at 1:26
  • curl: try 'curl --help' or 'curl --manual' for more information.. This is the message i get when i make those changes Commented May 25, 2012 at 1:42
  • The right way to do this is not to be using any shell commands at all. Python can do all of what curl, cut, sort, grep and wc do for you. Commented May 25, 2012 at 3:49

3 Answers 3

Reset to default
6

The right way to string together commands -- if you want to keep the shell out of it, which you should -- is to create multiple Popen objects.

def activeHostsQuery():
    args1 = ['curl', '-s', '-k',
             '-H', 'X-Requested-With: curl demoapp',
             '-u','username:password',
             'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv']
    args2 = ['cut', '-d', '-f1']
    args3 = ['sort', '-u']
    args4 = ['grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""']
    args5 = ['wc', '-l']

    p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
    p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE); p1.stdout.close()
    p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE); p2.stdout.close()
    p4 = subprocess.Popen(args4, stdin=p3.stdout, stdout=subprocess.PIPE); p3.stdout.close()
    p5 = subprocess.Popen(args5, stdin=p4.stdout, stdout=subprocess.PIPE); p4.stdout.close()
    activeHostsNumber = p5.communicate()[0]
    return activeHostsNumber

The advantage of this is that there's no shell involved -- you can substitute arbitrary variables into your argument lists without concern that they'll be string-split, misinterpreted, cause redirections, or anything else, and the distinctions between arguments you use in generating your lists will be honored.

Now, in this particular case, I'd do the whole thing in native Python -- there's no reason even to use curl when you have native HTTP libraries -- but knowing how to build pipelines with subprocess.Popen is useful in any event.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

By the way -- I'm pretty sure the regex given to grep here is wrong. I didn't change it because it was given as part of the problem, but you probably want to strip one of the sets of double quotation marks surrounding, unless you really want egrep to require the matched string to start and end with "".
imho, this is the best way to do something like this in python.
0

I would try this:

def activeHostsQuery():
    args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']

    activeHostsNumber = subprocess.Popen(" ".join("'%s'" % a for a in args), shell=True, stdout=subprocess.PIPE).communicate()[0]
    return activeHostsNumber

Edit: added quotes around arguments.

Another edit: Ok, try making the command a single string:

def activeHostsQuery():
    cmd = 'curl -s -k -H \'X-Requested-With: curl demoapp\' -u username:password \'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv\' | cut -d, -f1 | sort | uniq | grep -E \'"[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}"\' | wc -l'

    ctiveHostsNumber = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE).communicate()[0]
    return activeHostsNumber
Improve this answer

14 Comments

after making those adjustments it just returns a blank line and the program stops execution
Is it possible that i am not putting commas around the proper things in my args?
I don't think you need commas around anything. I made a change to my solution to add quotes around everything, since some of your arguments have spaces and special characters in them.
so i can just take all the commas and single quotes out of all the args?.. im trying that right now but not having much success
The args variable you have prepared is a list of strings. Each string must be surrounded by quotes and separated from the next one by a comma. But you don't want to pass a list of strings to subprocess.Popen when you're using the shell=True argument, you want to pass it a single string - which is why I have the " ".join("'%s'" % a for a in args) in my solution.
|
0

I know this doesn't answer the question... but that's shell script. If you want shell script pass an argument to sh -c (or bash or something)

    args = ['sh', '-c', 'curl -s -k -H X-Requested-With: curl demoapp -u','username:password https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv | cut -d -f1 | sort | uniq | grep -E "\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\"" | wc -l'


   count = int(cubprcess.check_output(args))

or use shell=True like some others suggested. This will definitely not work on Windows if you care about such things.

really you should probably do something like:

import requests
from csv
from StringIO import StringIO
import re

req=reqeusts.get(
    'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv',
    auth=('username','passoword'),
    headers={'X-Requested-With': 'curl demoapp'})

reader = csv.reader(StringIO(req.text))
count = 0
for line in reader:
    if re.match(r'.*\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*',line[0]) is not None:
        count += 1

print count
Improve this answer

1 Comment

sh -c "one line" "another line" doesn't actually process anything but the first line. The argument after -c is the script to run; the one after that is the $0 passed to the script, the one after that is $1, etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.