Profiling memory usage in python generators

Question

I'm trying to figure out where my memory is being used in a generator function in python 3. I've tried memprof and memory_profiler, and both fail to give me information on generators.

How do I determine where, and how much, memory I am allocating inside a generator function?

David Loughnane · Accepted Answer · 2019-07-31 10:40:07Z

1

The following snippet should work in python3. If you replace the generator function with a list you will see a large difference in memory allocation.

from memory_profiler import memory_usage

print(f'Memory usage after: {memory_usage()}MB')

def obj_generator(num_objects):
    for i in range(num_objects):
        new_obj = {
            'id' : i,
        }
        yield new_obj

objects = obj_generator(1_000_000)

print(f'Memory usage before: {memory_usage()}MB')

edited Jul 31, 2019 at 10:40

answered Jun 27, 2016 at 9:32

David Loughnane

1631 gold badge3 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Profiling memory usage in python generators

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related