Grabbing output FILE from Python Popen process?

Question

I have written a python program to interface with a compiled program (call it ProgramX) that has some idiosyncrasies that are proving difficult to deal with. I need to feed many thousands of input files to ProgramX via my python program. What I would like to do is to grab the output file that ProgramX creates with each run, and rename it something sensible, like inputfilename.output.

The problem comes in the output file that is written by ProgramX -- it is named via an unpredictable method, which will write, and "mercilessly overwrite", the output file if it already exists (which is the case the majority of the time). The saving grace probably comes with the fact that there is a standard prefix to the output files: think ProgramX.notQuiteRandomNumber.

The only think I can think to do is something like this in my bash shell:

PROGRAMXOUTPUT=$(ls -ltr ProgramX* | tail -n -1 | awk '{print $8}')
mv $PROGRAMXOUTPUT input.output

Which does 90% of what I need, but before I program all that bash into a series of Popen statements, is there a better way to do this? This problem feels like something people might have a much better solution than what I'm thinking.

Sidenote: I can grab the program's standard output without problems, however it's the output file that I need to grab.

Bonus: I was planning on running a bunch of instantiations of the program in the same directory, so my naive approach above may start to have unforeseen problems. So perhaps something fancy that watches the PID of ProgramX and follows its output.

Nicholas Riley · Accepted Answer · 2011-04-06 16:55:07Z

2

To do what your shell script above does, assuming you've only got one ProgramX* in the current directory:

import glob, os

programxoutput = glob.glob('ProgramX*')[0]
os.rename(programxoutput, 'input.output')

If you need to sort by time, etc., there are ways to do that too (look at os.stat), but using the most recent modification date is a recipe for nasty race conditions if you'll be running multiple copies of ProgramX concurrently.

I'd suggest instead that you create and change to a new, perhaps temporary directory for each run of ProgramX, so the runs have no possibility of treading on each other. The tempfile module can help with this.

edited Apr 6, 2011 at 16:55

answered Apr 5, 2011 at 22:08

Nicholas Riley

44.5k6 gold badges107 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alex · Accepted Answer · 2011-04-05 22:08:36Z

2

Two options that I see:

You could use lsof to find open files to find the files that ProgramX is writing.
A different approach would be to run ProgramX in a temporary directory (see tempfile for an easy way of setting up directories. Between runs of ProgramX, you can clean that directory or keep requesting new temp directories, if you are planning on running multiple copieProgramX at the same time.

answered Apr 5, 2011 at 22:08

Alex

3134 silver badges10 bronze badges

Comments

SiegeX · Accepted Answer · 2011-04-05 22:07:35Z

0

If there is only one ProgramX* file, then what about just:

mv ProgramX* input.output

answered Apr 5, 2011 at 22:07

SiegeX

141k25 gold badges150 silver badges159 bronze badges

Collectives™ on Stack Overflow

Grabbing output FILE from Python Popen process?

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related