Python: read output script twice and write two columns to csv

Question

This is my first python (2.4) program and I could use your help. I have a csv file

HOST, PATH
server1, /path/to/file1.py
server2, /path/to/file2.py
server3, /path/to/file3.py

That executes a command to each PATH on column 2/row[1]

In the output of the command I search for text and append them to new columns.

Example command output:

Command: python /path/to/file1.py
Output: server1 (NTFS) Reply:Yes

I would like make my new csv file like:

HOST, PATH, PLATFORM, REPLY
server1, /path/to/file1.py, Windows, Yes
server2, /path/to/file2.py, Linux, Yes
server3, /path/to/file3.py, BSD, No

I havnt yet got it to work with reading the same output line and appending two columns with different results. I have tried closing the writer and opening a new one within the same reader with no joy. I have tried indenting with no luck. The closest I have gotten is having it accurately search for both strings but it only writes to one column.

I have also tried first searching for the os platform and writing the changes to a new file Then, opening the new file to write another row

Of course I could force this to work by running the commands again and searching separately, but thats redundant and unnecessary.

import datetime
import csv
import os, time
from stat import * # ST_SIZE etc
from subprocess import Popen, PIPE, STDOUT

# Set Date
now = datetime.datetime.now()
today = now.strftime("%m-%d-%Y")

# Files
filename = "my_list.csv"
results = "results/results_" + today + ".csv" # Eg. results_04-14-2012.csv

# Commands
command = "python"
SP = " "

incsv = open(filename, 'rb')
try:
    reader = csv.reader(incsv)

    outcsv = open(results, 'w')
    try:
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(command + SP + row[1], shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (command + SP + row[1], stdout)

            if not stdout:
                writer.writerow(row + ['PLATFORM']) # Create header for new column when first line is empty on stdout
            elif 'BSD' in stdout:
                writer.writerow(row + ['BSD'])
            elif 'Linux' in stdout or 'swap' in stdout or 'LLVM' in stdout or 'EBR' in stdout:
                writer.writerow(row + ['Linux'])
            elif 'NTFS' in stdout:
                writer.writerow(row + ['Windows'])
            else:
                writer.writerow(row + ['Error Scanning'])

                    reply = open(results, 'w')
                    try:
                        writer = csv.writer(platform)

                        for row in reader:

                            if not stdout:
                                writer.writerow(row + ['REPLY']) # Create header for new column when first line is empty on stdout
                            elif 'Reply:Yes' in stdout:
                                writer.writerow(row + ['Yes'])
                            elif 'Reply:No' in stdout :
                                writer.writerow(row + ['No'])

                            else:
                                writer.writerow(row + ['Error'])

                    finally:
                        reply.close()
    finally:
        outcsv.close()
finally:
    incsv.close()

alexis · Accepted Answer · 2012-04-16 14:27:15Z

1

Collect all the information you want from the output, and write it out using a single call to writerow(). E.g.,

newdata = ['BSD', 'Yes']
...
writer.writerow(row + newdata)

Of course you will be building up newdata in steps, by parsing the script results.

PS. Update your python! 2.4 is way too old. Updating will make your life easier in countless ways.

edited Apr 16, 2012 at 14:27

answered Apr 16, 2012 at 14:22

alexis

50.4k18 gold badges107 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Tommy Over a year ago

Unfortunately this is a production server that I cannot upgrade. Thanks for your help, I'll see where that gets me.

Tommy Over a year ago

I thought it might be an easy fix, and that it was. Very grateful, thanks alexis

vsekhar · Accepted Answer · 2012-04-14 19:38:08Z

1

You may not need csv if you have a 1:1 mapping between input lines and output lines. Just process each line and spit out a new one.

Method 1: Pipes

import sys
import subprocess

def parse_csv(input, output):
    # echo headings with new columns
    print >> output, input.readline() + ', PLATFORM, REPLY'

    for line in input.readlines():
        server, _, path = line.partition(',')
        path = path.strip()
        p = subprocess.Popen(command + ' ' + path,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             shell=True)
        stdout, _ = p.communicate()

        # your additional logic goes here based on contents of stdout

        # when ready to output, just print, e.g.
        print >> output, '%s, %s, %s' % (line, stdout, 'Yes')

if __name__ == '__main__':
    parse_csv(sys.stdin, sys.stdout)

Execute with

$ cat servers_and_paths.csv | python add_two_columns.py > servers_paths_and_flags.csv

Method 2: Files

If you don't want to use pipes, use with clauses to handle opening/closing:

from contextlib import closing

if __name__ == '__main__':
    with closing(open('servers_and_paths.csv')) as input,
         closing(open('servers_paths_and_flags.csv', 'wb')) as output:
        func_name(input, output)

answered Apr 14, 2012 at 19:38

vsekhar

5,6005 gold badges24 silver badges24 bronze badges

2 Comments

Tommy Over a year ago

Im running 2.4 and am unable to upgrade which is why im not using with statements. wish i could, ive seen many examples using them

Tommy Over a year ago

I like the concept of this, I nearly got it working after making it 2.4 compliant and will continue to test it. Thanks for your input vsekhar

Collectives™ on Stack Overflow

Python: read output script twice and write two columns to csv

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related