How to write the output of an executable binary into memory rather than disk

Question

I got a binary that works like the below:

> ./my_bin raw.avi output_file.avi

output_file.avi is what I want, some verbose information will print in the terminal when the job is succeeded, like:

Copyright 2022 Company Inc... Success.

I want to run this command inside my code and redirect the output_file.avi into some byte array so that I don't have to read it from disk and delete it. My approach looks like the below Golang snippet:

func wrongOne(stdin []byte) ([]byte, error) {
    inBuf := bytes.NewBuffer(stdin)
    outBuf := bytes.NewBuffer(nil)
    cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
    cmd.Stdin = inBuf
    cmd.Stdout = outBuf
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    return outBuf.Bytes(), nil // wrong
}

However, the return byte array is longer than the below approach, which leads to failure on the MD5 check.

func correctOne(stdin []byte) ([]byte, error) {
    inBuf := bytes.NewBuffer(stdin)
    cmd := exec.Command("./my_bin", "/dev/stdin", "output_file")
    cmd.Stdin = inBuf
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    return os.ReadFile("output_file")
}

the wrongOne function can be modified to following code to be correct:

func modifiedWrongOne(stdin []byte) ([]byte, error) {
    inBuf := bytes.NewBuffer(stdin)
    outBuf := bytes.NewBuffer(nil)
    cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
    cmd.Stdin = inBuf
    cmd.Stdout = outBuf
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    correct, _ := correctOne(stdin)
    return outBuf.Bytes()[:len(correct)], nil // diff
}

I presume that the output verbose information is included in the /dev/stdout so that the wrongOne function doesn't works. i.e.,

the output of wrongOne = the output of correctOne + []byte{"Copyright 2022 Company Inc... Success."}

Is there any solution that I can get the output_file.avi in the pipe without save it as file and read it from disk? Thanks!

the quick hack is : output = bytes.TrimSuffix(output, []byte{"Copyright 2022 Company Inc... Sucess"}) (possibly with a "\n" somewhere) — LeGEC
– LeGEC, Commented Aug 31, 2022 at 5:57
@LeGEC sadly, the verbose information will contain certain information related to the input, trim with static string won't work since it changes from time to time :( — hide1nbush
– hide1nbush, Commented Aug 31, 2022 at 6:10

score 4 · Accepted Answer · 2022-08-31 06:17:24Z

4

The command writes the copyright notice to stdout. To avoid commingling the copyright notice with the output file, use a file other than /dev/stdout as the output file.

The function below uses Cmd.ExtraFiles to connect a pipe to fd 3 in the child process. The function copies data from the pipe to a byte buffer and returns those bytes to the caller.

func otherOne(stdin []byte) ([]byte, error) {
    r, w, err := os.Pipe()
    if err != nil {
        return nil, err
    }
    defer r.Close()
    defer w.Close()

    cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/fd/3")
    cmd.Stdin = bytes.NewReader(stdin)
    cmd.ExtraFiles = []*os.File{w} // The first file is fd 3.
    if err := cmd.Start(); err != nil {
        return nil, err
    }
    w.Close()
    var outbuf bytes.Buffer
    if _, err := io.Copy(&outbuf, r); err != nil {
        return nil, err
    }
    if err := cmd.Wait(); err != nil {
        return nil, err
    }
    return outbuf.Bytes(), nil
}

edited Aug 31, 2022 at 6:17

answered Aug 31, 2022 at 5:50

user5728991

Sign up to request clarification or add additional context in comments.

2 Comments

hide1nbush Over a year ago

Thanks for your detailed answer. Here's another question: is this function thread-safe? We may have several requests to call this program at the same time. Will they have conflicts on writing/reading something like /dev/fd/3 or /dev/stdin?

LeGEC Over a year ago

@WillZhao: exec.Command(...) will create a whole new process, /dev/stdin and /dev/fd/3 is defined for each process. So yes : you can run these concurrently.

hide1nbush · Accepted Answer · 2022-11-08 07:21:58Z

0

After months, I figure out another way to solve this problem. The basic idea is similar with Cerise, i.e., using /dev/fd/3 to redirect the output file. After that, we redirect /dev/fd/3 to /dev/stdout, verbose log to /dev/stderr by 3>&1, 1>&2, respectively. An additional gen.sh is added. Here's the solution:

#gen.sh
./mybin /dev/stdin /dev/fd/3 3>&1 1>&2

// gen.go
func gen(stdin []byte) ([]byte, error) {
    inBuf := bytes.NewBuffer(stdin)
    outBuf := bytes.NewBuffer(nil)
    cmd := exec.Command("./gen.sh")
    cmd.Stdin = inBuf
    cmd.Stdout = outBuf
    cmd.Stderr = os.Stderr
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    return outBuf.Bytes(), nil
}

edited Nov 8, 2022 at 7:21

answered Nov 8, 2022 at 7:12

hide1nbush

9451 gold badge4 silver badges16 bronze badges

Collectives™ on Stack Overflow

How to write the output of an executable binary into memory rather than disk

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related