Running external python in Golang, Catching continuous exec.Command Stdout

Question

So my go script will call an external python like this

cmd = exec.Command("python","game.py")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
go func(){
    err := cmd.Run()
    if err != nil{
    panic(err)
    }
}()

It runs my python script concurrently which is awesome. But now the problem is, my python script will run infinitely and it will print out some information from time to time. I want to "catch" these Stdout and print them out on my golang terminal. How do I do it concurrently (without waiting my python script to exit)?

Community · Accepted Answer · 2020-06-20 09:12:55Z

8

Use cmd.Start() and cmd.Wait() instead of cmd.Run().

https://golang.org/pkg/os/exec/#Cmd.Run

Run starts the specified command and waits for it to complete.

Start starts the specified command but does not wait for it to complete.

Wait waits for the command to exit. It must have been started by Start.

And if you want to capture stdout/stderr concurrently, use cmd.StdoutPipe() / cmd.StderrPipe() and read it by bufio.NewScanner()

package main

import (
    "bufio"
    "fmt"
    "io"
    "os/exec"
)

func main() {
    cmd := exec.Command("python", "game.py")
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        panic(err)
    }
    stderr, err := cmd.StderrPipe()
    if err != nil {
        panic(err)
    }
    err = cmd.Start()
    if err != nil {
        panic(err)
    }

    go copyOutput(stdout)
    go copyOutput(stderr)
    cmd.Wait()
}

func copyOutput(r io.Reader) {
    scanner := bufio.NewScanner(r)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
}

The following is a sample python code for reproducing real-time output. The stdout may be buffered in Python. Explicit flush may be required.

import time
import sys

while True:
    print "Hello"
    sys.stdout.flush()
    time.sleep(1)

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jan 1, 2017 at 12:29

minamijoyo

3,4531 gold badge21 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Louis Ng Over a year ago

Thanks! However this still doesn't help with capturing standard output of the cmd.

minamijoyo Over a year ago

Oh, I'm sorry. if you want to capture stdout concurrently, how about cmd.StdoutPipe()?

Louis Ng Over a year ago

Hi, Thanks again. I tried the new code but the code block at for scanner.Scan() and will not readline until my python script exit.

minamijoyo Over a year ago

I don't know about the implementation of game.py , but stdout may be bufferd inside Python. How about explicitly specifying sys.stdout.flush() in Python?

minamijoyo Over a year ago

Congrats. I added the sample reproduction python code to the answer.

|

hyman zhang · Accepted Answer · 2021-07-22 08:53:49Z

1

You can use add the flag -u to turn off buffering,

cmd := exec.Command("python", "-u", "game.py")

https://docs.python.org/3/using/cmdline.html#cmdoption-u

https://bugs.python.org/issue526382

edited Jul 22, 2021 at 8:53

user4466350

answered Jul 22, 2021 at 8:19

hyman zhang

1211 silver badge3 bronze badges

Comments

Damien D · Accepted Answer · 2022-06-08 13:27:18Z

1

If you want to capture stdout and stderr concurrently, indeed you need to use cmd.StdoutPipe(), cmd.StderrPipe(), cmd.Start() and cmd.Wait().

But as mentioned in the documentation (https://pkg.go.dev/os/exec#Cmd.StdoutPipe):

It is thus incorrect to call Wait before all reads from the pipe have completed.

You'll need to wait for you goroutines to finish before calling cmd.Wait() (see the following thread: https://github.com/golang/go/issues/38268)

answered Jun 8, 2022 at 13:27

Damien D

111 bronze badge

Collectives™ on Stack Overflow

Running external python in Golang, Catching continuous exec.Command Stdout

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related