Converting a bash script into Python using subprocess module

Question

I am trying to convert a bash script into a subprocess so I can schedule different parameters. Here is my original bash script:

#!/bin/sh

set -xe

export NVIDIA_VISIBLE_DEVICES=0
export CUDA_VISIBLE_DEVICES=0
export TF_CUDNN_RESET_RND_GEN_STATE=1

python3 -u DeepSpeech.py \
  --train_files /external_data/data_csvs/train.csv \
  --test_files  /external_data/data_csvs/test.csv \
  --dev_files  /external_data/data_csvs/dev.csv \
  --epochs 30 \
  --train_batch_size 32 \
  --dev_batch_size 32 \
  --test_batch_size 32 \
  --export_dir /external_data/deepspeech_models/ \
  --use_allow_growth  \
  --n_hidden 2048 \
  --train_cudnn  \
  --learning_rate 0.00005 \
  --dropout_rate 0.40 \
  --summary_dir /external_data/tensorboard_summaries/ \
  --checkpoint_dir /external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/ | tee //tmp/external/deepspeech_models/progress.txt \
  "$@"

Now I am trying to convert this into a subprocess with the following:

    subprocess.Popen([
        'set', '-xe',
        'export', 'NVIDIA_VISIBLE_DEVICES=0',
        'export', 'CUDA_VISIBLE_DEVICES=0',
        'export', 'TF_CUDNN_RESET_RND_GEN_STATE=1',
        'python3', '-u', 'DeepSpeech.py',
        '--train_files', '/external_data/data_csvs/train.csv',
        '--test_files'  '/external_data/data_csvs/test.csv',
        '--dev_files', '/external_data/data_csvs/dev.csv',
        '--epochs', str(epochs),
        '--train_batch_size', str(trainbs),
        '--dev_batch_size', str(devbs),
        '--test_batch_size', str(testbs),
        '--export_dir', '/external_data/deepspeech_models/',
        '--use_allow_growth',  
        '--n_hidden', str(2048),
        '--train_cudnn',  
        '--learning_rate', str(lr),
        '--summary_dir', '/external_data/tensorboard_summaries/' 
        '--checkpoint_dir', '/external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/', '|', 'tee', '/external_data/deepspeech_models/Deepspeech_progress.txt',
        '$@'], shell = True, cwd = '//DeepSpeech/', stdout = subprocess.PIPE, executable = '/bin/sh')

the str() values are just values I am using as variables into my subprocess.

It runs without error but nothing happens. Am I missing something? The bash script config runs fine. Also, how would I go about getting output processed to stdout when I run the script?

What output are you expecting? Can you see the process in top? — Dallan
– Dallan, Commented Aug 7, 2020 at 19:55
@Dallan I am training a neural network using tensorflow. I should be seeing some model progress and some steps being taken. The process is not in top Perhaps I need to add the '\' into my subprocess ? — Coldchain9
– Coldchain9, Commented Aug 7, 2020 at 19:56
You are missing the ; between the individual commands. Right now, everything is being passed as arguments to set — Dallan
– Dallan, Commented Aug 7, 2020 at 19:58
@Dallan Of course! Ok. would those go as separate ';' tokens in my list of arguments? — Coldchain9
– Coldchain9, Commented Aug 7, 2020 at 20:00
Instead of "export VAR1=val1; export VAR2=val2; command arg1 arg2", shell=True, you say: ["command", "arg1", "arg2"], env={"VAR1": "val1", "VAR2": "val2"} and you can happily leave shell out of it with default False. Besides even in shell itself, when you write a one-liner, you can export variables for a single command by prepending them to its command line. I.e. VAR1=val1 VAR2=val2 command arg1 arg2 instead export which has to be separate command terminated by newline or ;. — Ondrej K.
– Ondrej K., Commented Aug 7, 2020 at 20:51

Dallan · Accepted Answer · 2020-08-07 20:06:39Z

1

As in this answer, you need to separate the individual commands with ;.

In your case, try something like:

train_cmd = ['python3', '-u', 'DeepSpeech.py',
            '--train_files', '/external_data/data_csvs/train.csv',
            '--test_files'  '/external_data/data_csvs/test.csv',
            '--dev_files', '/external_data/data_csvs/dev.csv',
            '--epochs', str(epochs),
            '--train_batch_size', str(trainbs),
            '--dev_batch_size', str(devbs),
            '--test_batch_size', str(testbs),
            '--export_dir', '/external_data/deepspeech_models/',
            '--use_allow_growth',  
            '--n_hidden', str(2048),
            '--train_cudnn',  
            '--learning_rate', str(lr),
            '--summary_dir', '/external_data/tensorboard_summaries/' 
            '--checkpoint_dir', '/external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/', '|', 'tee', '/external_data/deepspeech_models/Deepspeech_progress.txt',
            '$@']
cmds = ['set -xe', 
        'export NVIDIA_VISIBLE_DEVICES=0', 
        'export CUDA_VISIBLE_DEVICES=0',
        'export TF_CUDNN_RESET_RND_GEN_STATE=1',
        ' '.join(train_cmd)]
subprocess.Popen('; '.join(cmds), shell = True, cwd = '//DeepSpeech/', stdout = subprocess.PIPE, executable = '/bin/sh')

answered Aug 7, 2020 at 20:06

Dallan

5164 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Coldchain9 Over a year ago

How come in my original bash script, I didn't need the semicolons? Pretty new to bash scripting so any help is appreciated.

Dallan Over a year ago

@Coldchain9 The bash script simply puts each command into the shell sequentially, splitting on newlines (the \ escapes the newlines in your python command). If you wanted to run it in the shell with one command, you would also have to join them with semicolons. See: docstore.mik.ua/orelly/unix3/upt/ch28_16.htm

Coldchain9 Over a year ago

This is all incredibly helpful. It is now starting and completes some of the starting process but then I get a BrokenPipeError

Coldchain9 Over a year ago

Nevermind. I fixed it by removing the stdout argument. IT is now running and working. Thank you a ton for your help. I will accept as the answer soon.

Collectives™ on Stack Overflow

Converting a bash script into Python using subprocess module

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related