Run bash commands in parallel using iPython

Question

I'm using iPython notebook on a 16 core machine (so there are 15 "engines"). If I run this cell ...

%%px
%%bash
echo 'hi' > file1.txt

... then the result is one file called file1.txt that is written to disk 15 times. What I really want is to write to 15 different files, file1.txt through file15.txt.

I'm new to all this, so I imagine there's a simple solution!

Thanks,

Retsreg

John Zwinck · Accepted Answer · 2015-01-20 01:43:16Z

1

The Bash special variable $$ is the PID of the shell, so you could do this:

echo hi > file$$.txt

That will write files like file3392.txt etc., where the number is the PID of each session. You can later rename the files if you need to.

answered Jan 20, 2015 at 1:43

John Zwinck

252k44 gold badges346 silver badges459 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

retsreg Over a year ago

Comfirmed that this works, many thanks! Is there a similar way to get the date and time in the filename too?

John Zwinck Over a year ago

You can try file$$_$(date +%F_%T).txt. The $() evaluates its argument as a command and captures the output string, and you can use whatever format you want in the date command (I chose a standard one).

Collectives™ on Stack Overflow

Run bash commands in parallel using iPython

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related