Piping output from a python script to a remote machine?

Question

I have a python script that output/prints a large array (50MB in size) to the screen. This script is running on machineA. Is it at all possible to 'pipe' this output to machineB, where it will be processed?

I know I can save the output in a file on machineA and then send the file to machineB. I was just wondering if it is possible to do so without having to save the data first to a file (on machineA).

Any solutions (python, bash shell, other ideas) will be appreciated.

6502 · Accepted Answer · 2011-03-11 20:43:17Z

6

This is a work for nc (netcat)... on machineB you do

nc -l 12345 | processing_program

This command will start netcat in "listen mode" waiting for a connection. Once the connection is established what is coming from the network will be sent to stdout and what is sent to stdin will be sent back to whoever connected.

After that you go on machineA and run

generating_program | nc machineB 12345

this will instruct netcat to start a connection to machineB (port 12345) and send whatever it gets from stdin over the wire. Whatever comes back from that connection is sent to standard output.

answered Mar 11, 2011 at 20:43

6502

115k17 gold badges177 silver badges277 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

JCCyC Over a year ago

Your answer is better (no need for passwords, or for running a service).

user3262424 Over a year ago

6502: thank you. can machineB be listening to 20 machines at the same time? also, must I actively 'open the listening port' on machineB?

6502 Over a year ago

With that command machineB will be listening to whoever connects (so this is of course not secure at all... anyone on the internet without any authentication can send data to processing_program using machineB if that machine can be reached!). nc terminates once the client closes the connection so you will eventually need to rerun the command in a loop. If for "opening the listening port" you mean making an hole in your firewall or defining a virtual server path in your router the answer is yes. nc simply does a TCP listen on the specified port.

Ryan C. Thompson · Accepted Answer · 2011-03-11 20:43:42Z

2

On machineA:

python script-on-machineA.py --options | ssh machineB tee /file/on/machine/B.txt

Alternatively, on machineB:

ssh machineA python /path/to/script-on-machineA.py --options | cat > /file/on/machine/B.txt

answered Mar 11, 2011 at 20:43

Ryan C. Thompson

42.4k30 gold badges103 silver badges164 bronze badges

1 Comment

user3262424 Over a year ago

This would work well, since I can easily SSH between the machines.

ChrisJ · Accepted Answer · 2011-03-11 20:42:40Z

1

With the Netcat tool you may easily pipe data from one machine to another.

answered Mar 11, 2011 at 20:42

ChrisJ

5,28127 silver badges20 bronze badges

Comments

nhed · Accepted Answer · 2011-03-11 20:43:09Z

1

You can do it on different levels - here are a few options

use ssh to pipe myprog | ssh remotemachine myotherprog
use nfs (if going to a file
use netcat (nc)
use something like thrift

It depends on how solid & permanent the solution needs to be

answered Mar 11, 2011 at 20:43

nhed

6,0073 gold badges33 silver badges46 bronze badges

Comments

JCCyC · Accepted Answer · 2011-03-11 20:43:10Z

1

Maybe something along the lines of

myPythonScript | ssh user@machine "sh -c 'cat >myfile.txt'"

WARNING: pulled entirely out of my head and untested!

answered Mar 11, 2011 at 20:43

JCCyC

16.8k11 gold badges51 silver badges78 bronze badges

Collectives™ on Stack Overflow

Piping output from a python script to a remote machine?

5 Answers 5

3 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related