How to get the max memory usage of a program using psutil in Python

Question

I am using the following code to get the max memory usage of the program.

    import os, subprocess , psutil
    def mem(cmd):
        try:
            with open('in.txt','r') as infile, open('out.txt', 'w') as outfile:
                p=psutil.Popen("./"+cmd,shell=False,stdin=infile,stdout = outfile)
            print p.memory_info()
        except Exception:
             print "Error"
    cmd=raw_input()
    mem(cmd)

The problem is sometimes for initial runs of the program the memory usage output is (0,0) but subsequently it displays the correct output. I dont why this happening. For some programs such as the hello world program in c++ the output is pmem(rss=4096, vms=315392) which is about 0.3M( I think the output is in bytes ) but running the hello world program in ideone.com gives the output as ~3M. Why is there this disperency?

cmd is the name of the executable.

The output of command print subprocess.check_output(['ps', 'v', '-p', str(p.pid)])

PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 16150 pts/16 Z+ 0:00 0 0 0 0 0.0 [a.out] <defunct>

One of my sample C++ program:

`int a[1000000];
int main()
{
    return 0;
}`

returns pmem(rss=4096, vms=4313088) sometimes and pmem(rss=0,vms=0) sometimes

Could you reproduce it with a known program such as ["python", "-c", "pass"]? Are you sure, you haven't suppressed an exception i.e., Popen() started successfully? — jfs
– jfs, Commented Mar 29, 2014 at 15:37
unrelated: if stderr=PIPE then you should read from p.stderr otherwise the process may block forever if it generates enough output on stderr to fill the corresponding OS pipe buffer. — jfs
– jfs, Commented Mar 29, 2014 at 15:38
What does print subprocess.check_output(['ps', 'v', '-p', str(p.pid)]) show? — jfs
– jfs, Commented Mar 29, 2014 at 15:43
@J.F.Sebastian I added exception handling . Sometime the output is (0,0) and there is no exception — user2179293
– user2179293, Commented Mar 29, 2014 at 15:45

mdadm · Accepted Answer · 2014-03-29 16:15:13Z

3

The problem here is that psutils takes a quick snapshot from the /proc filesystem, as you can see in the source.

When you run your hello world example, in some cases it finishes before python gets a chance to read the values from /proc.

Once the process is finished, it effectively no longer consumes any ram. You can confirm this with an strace.

open("/proc/13420/statm", O_RDONLY)     = 3
read(3, "0 0 0 0 0 0 0\n", 1024)        = 14

If you modify your example to use something like sleep, you'll notice that psutils consistently gives memory usage back.

#include <iostream>
#include <unistd.h>

int main()
{
  std::cout << "Hello World.. sleeping!";
  sleep(3);
}

Output of your python script...

a.out
meminfo(rss=286720, vms=12931072)

One simple way to accomplish what you are trying to do, is by using the /usr/bin/time command, which on most platforms will give you the average total memory usage of the process you launch OR use valgrind as J.F Sebastian suggests... whom posted as I was researching and testing my answer ;)

Hello World.. sleeping!0.00user 0.00system 0:03.00elapsed 0%CPU     
(0avgtext+0avgdata 1144maxresident)k
0inputs+0outputs (0major+348minor)pagefaults 0swaps

answered Mar 29, 2014 at 16:15

mdadm

1,3631 gold badge12 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

jfs Over a year ago

/usr/bin/time -f %M <command> will print "Maximum resident set size of the process during its lifetime, in Kilobytes"

user2179293 Over a year ago

Why rss size is so low when I am running the program. My max rss size is 3292KB for a program whose actual memory usage is about 49MB? Is there a problem with my system?

Community · Accepted Answer · 2017-05-23 12:16:10Z

2

<defunct> means that the subprocess is a zombie process (it is dead but its status has not been read yet by the parent (p.poll() or p.wait())). It seems both psutil and ps shows RSS to be zero for such processes.

The result depends on whether the subprocess will exit sooner than p.memory_info() is called. It is a race. If you add a delay at the exit in the C++ program then p.memory_info() may be called before the subprocess exits and you should get non-zero results.

The problem is that I can get arbitrary programs to evaluate . The language is also not fixed. Isn't there an elegant solution to this problem?

You might need OS support to save info about the memory usage by a subprocess even after it exits. Or you could run the program using a memory profiler such as valgrind and read its results. To collect results:

$ valgrind --tool=massif cmd arg1 arg2

To see the results, you could use ms_print:

$ ms_print massif.out.* | less

Or GUI Massif-Visualizer

@mdadm suggested a simpler solution: time command:

from subprocess import Popen, PIPE

p = Popen(['time', '-f', '%M'] + args, stderr=PIPE)
ru_maxrss = int(p.communicate()[1])
print("Maximum rss %d KB" % ru_maxrss)

GNU time uses wait3() to populate resource usage info if it is available. It can be called in Python:

import os
from subprocess import Popen

p = Popen(args)
ru = os.wait4(p.pid, 0)[2]
print("Maximum rss %d KB" % ru.ru_maxrss)

I've compared the maximum value returned by psutil.Process.memory_info (rss) with ru_maxrss value returned by os.wait4 and with the maximum total memory reported by valgrind --tool=massif: they are similar.

11 Comments

user2179293 Over a year ago

but I got a <defunct> even when pmem(rss=4096, vms=4313088)... Is it due to race conditions because I put the print statement after p.memory_info()

jfs Over a year ago

The subprocess may have already died by the time you called ps but it was still alive during p.memory_info()

user2179293 Over a year ago

The problem is that I can get arbitrary programs to evaluate . The language is also not fixed. Isn't there a elegant solution to this problem

user2179293 Over a year ago

i tried using /usr/bin/time -v ./a.out<int.txt>out.txt but still the resident set size is not accurate, it is very low

jfs Over a year ago

I've added os.wait4()-based solution. It is similar to your original resource-based solution. Have you compared them to psutil, valgrind results?

|

Collectives™ on Stack Overflow

How to get the max memory usage of a program using psutil in Python

2 Answers 2

2 Comments

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related