using subprocess over different files python

Question

I've got a problem with a short script, it'd be great if you could have a look!

import os
import subprocess

root = "/Users/software/fmtomov1.0/remaker_lastplot/source_relocation/observed_arrivals_loc3d"

def loop_loc3d(file_in):
    """Loops loc3d over the source files"""
    return subprocess.call (['loc3d'], shell=True)

def relocation ():
    for subdir, dirs, files in os.walk(root):
        for file in files:
            file_in = open(os.path.join(subdir, file), 'r')
            return loop_loc3d(file_in)

I think the script is quite easy to understand, it's very simple. However I'm not getting the result wanted. In a few word I just want 'loc3d' to operate over all the files contents present in the 'observed_arrivals_loc3d' directory, which means that I need to open all the files and that's what I've actually done. In fact, if I try to 'print files' after:

for subdir, dirs, files in os.walk(root)

I'll get the name of every file. Furthermore, if I try a 'print file_in' after

file_in = open(os.path.join(subdir, file), 'r')

I get something like this line for every file:

<open file '/Users/software/fmtomov1.0/remaker_lastplot/source_relocation/observed_arrivals_loc3d/EVENT2580', mode 'r' at 0x78fe38>

subprocess has been tested alone on only one file and it's working.

Overall I'm getting no errors but just -11 which means absolutely nothing to me. The output from loc3d should be completly different.

So does the code look fine to you? Is there anything I'm missing? Any suggestion?

Thanks for your help!

also consider using make as suggested in the comment by @Kirill Teplinskiy. — jfs
– jfs, Commented Oct 30, 2011 at 1:34
It's kind of hard to understand that for me. Anyway I tried what you said below and nothing happens (no output no error). At this stage I do think it's a 'loc3d' bug, so probably everything is fine as far as python is concerned. Will try to look deeper into loc3d. — eikonal
– eikonal, Commented Oct 30, 2011 at 1:53

unutbu · Accepted Answer · 2011-10-29 08:04:24Z

1

I assume you would call loc3d filename from the CLI. If so, then:

def loop_loc3d(filename):
    """Loops loc3d over the source files"""
    return subprocess.call (['loc3d',filename])

def relocation():
    for subdir, dirs, files in os.walk(root):
        for file in files:
            filename = os.path.join(subdir, file)
            return loop_loc3d(filename)

In other words, don't open the file yourself, let loc3d do it.

answered Oct 29, 2011 at 8:04

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

unutbu Over a year ago

Does loc3d filename work as expected from the command line? Does loc3d have a -v verbose mode? If so, calling subprocess.call (['loc3d','-v',filename]) might give you more debugging info.

eikonal Over a year ago

Yes it does work....with -v it still works but doesn't give any output now and no errors as well.

unutbu Over a year ago

Maybe loc3d relies on environment variables which are not present when called from subprocess.call. Does subprocess.Popen('loc3d filename', shell=True) work?

unutbu Over a year ago

If you know the environment variables loc3d needs, it is better not to use shell=True and instead supply the environment variables directly: subprocess.Popen(['loc3d',filename],env={'NAME':'VALUE',...})

eikonal Over a year ago

Popen gives the same result (i.e., none). I know what loc3d needs but it's a long list of files. And for example I'm trying to iterate it over 411 files. The strange thing is that it works for a file, but not for all of them.

Uku Loskit · Accepted Answer · 2011-10-29 08:16:55Z

1

Currently your relocation method will return after the first iteration (for the first file). You shouldn't need to return at all.

def loop_loc3d(filename):
    """Loops loc3d over the source files"""
    return subprocess.call (['loc3d',filename])

def relocation ():
    for subdir, dirs, files in os.walk(root):
        for file in files:
            filename = os.path.join(subdir, file)
            loop_loc3d(filename)

This is only one of the issues. The other is concerning loc3d itself. Try providing the full path for loc3d.

edited Oct 29, 2011 at 8:16

answered Oct 29, 2011 at 8:06

Uku Loskit

42.2k9 gold badges97 silver badges98 bronze badges

3 Comments

eikonal Over a year ago

It looks like it's working for a few seconds but then no output and no errors....

Uku Loskit Over a year ago

any idea where I can download loc3d to test it?

eikonal Over a year ago

unfortunately you can't and I think I can't attach anything here otherwise it would be a good idea!

jfs · Accepted Answer · 2011-10-29 21:59:02Z

0

-11 exit code might mean that the command killed by signal Segmentation fault. It is a bug in loc3d. A well-behaved program should not produce 'Segmentation fault' on any user input.

Feed loc3d only files that it can understand. Print filenames or use subprocess.check_call() to find out which file it doesn't like:

#!/usr/bin/env python
import fnmatch
import os
import subprocess

def loc3d_files(root):
    for dirpath, dirs, files in os.walk(root, topdown=True):
        # skip hidden directories
        dirs[:] = [d for d in dirs if not d.startswith('.')]
        # process only known files
        for file in fnmatch.filter(files, "*some?pattern[0-9][0-9].[ch]"):
            yield os.path.join(dirpath, file)

for path in loc3d_files(root):
    print path
    subprocess.check_call(['loc3d', path]) # raise on any error

answered Oct 29, 2011 at 21:59

jfs

417k210 gold badges1k silver badges1.7k bronze badges

2 Comments

eikonal Over a year ago

Thanks very much J.F., you're really close to the problem, I mean, very often I get 'Segmentation fault'. I'll try that as soon as I can and I'll let you know.

jfs Over a year ago

@eikonal: In the comment you said that nothing happens. If you are talking about the above code then you must change the file pattern "*some?pattern[0-9][0-9].[ch]" e.g., to feed all .txt-files to loc3d you could use "*.txt" file pattern. If you'd like to process files from directories that have names that start with a dot then you must remove dir[:] = ... line in the above code. You must see at least something in the output if loc3d_files() yields at least one filename.

eikonal · Accepted Answer · 2011-11-04 04:14:30Z

0

Just found out that loc3d, as unutbu said, relies on several variables and in the specific case one called 'observal_arrivals' that I have to create and delete every time from my directory. In Pythonic terms it means:

import os
import shutil
import subprocess

def loop_loc3d(file_in):
    """Loops loc3d over the source files"""
    return subprocess.call(["loc3d"], shell=True)

path = "/Users/software/fmtomo/remaker_lastplot/source_relocation"
path2 = "/Users/Programming/working_directory/2test"
new_file_name = 'observed_arrivals'
def define_object_file ():
    for filename in os.listdir("."):
        file_in = os.rename (filename, new_file_name) # get the observal_arrivals file
        file_in = shutil.copy ("/Users/simone/Programming/working_directory/2test/observed_arrivals", "/Users/software/fmtomo/remaker_lastplot/source_relocation")
        os.chdir(path) # goes where loc3d is
        loop_loc3d (file_in)
        os.remove("/Users/software/fmtomo/remaker_lastplot/source_relocation/observed_arrivals")
        os.remove ("/Users/Programming/working_directory/2test/observed_arrivals")
        os.chdir(path2)

Now, this is working very well, so it should answer my question. I guess it's quite easy to understand, it's just copying, changing dir and that kind of stuff.

edited Nov 4, 2011 at 4:14

answered Nov 1, 2011 at 7:56

eikonal

1711 gold badge4 silver badges13 bronze badges

5 Comments

jfs Over a year ago

1. Don't post questions as an answer. 2. There are many things wrong with this code. Read Python Tutorial. It will save you time. 3. open('fort.14', 'r') means: open file 'fort.14' for reading. It doesn't create it if the file doesn't exists.

eikonal Over a year ago

what should I do in this case? I mean I don't think it is a good idea to ask a new question and I don't think I can post that in a comment. Obviously fort.14 already exists.

jfs Over a year ago

Create a complete (i.e., you can run it as is), minimal (i.e., the problem vanishes if you remove anything from it) example that shows your problem and post it as a new question. You can add a link to your previous question if you think that it is related.

jfs Over a year ago

Add a complete traceback (lines that you see in the output before OSError: ...).

eikonal Over a year ago

I've sorted it out... you know what? Never programming during the night, you can make very stupid mistakes. Anyway I'm going to edit this last answer and leave it without the added question

Collectives™ on Stack Overflow

using subprocess over different files python

4 Answers 4

5 Comments

3 Comments

2 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

3 Comments

2 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related