If the type of X is numpy.array with dtype='f4'
Then what is the memory behavior of the line below:
X = array(X, dtype=double, copy=False, order='C', subok=True)
Does it allocate new memory space for X (Memory usage * 2)?
Add a comment
|
2 Answers
The NumPy doc has this to say about the copy argument to the array constructor:
If true (default), then the object is copied. Otherwise, a copy will only be made if array returns a copy, if obj is a nested sequence, or if a copy is needed to satisfy any of the other requirements (dtype, order, etc.).
You have provided an example of one of the "other requirements", dtype. So the answer is that in this case, copy=False is ignored, because a copy must be made to cast the float32 content into higher precision.
I like to run little experiments to see what "really" happens in my system. Here's a simple script:
import numpy as np
import numpy.random as rand
@profile
def test_size(N=20e6):
x = np.zeros(N, dtype=np.float32)
x[0] = 200
x[N-1] = 123.555
x[0:N] = rand.rand(N)
x = np.array(x, copy=False, dtype=np.double, order='C', subok=True)
x[0:N] = rand.rand(N)
y = np.zeros(N, dtype=np.float32)
y[0] = 500.0
y[1000] = 123.4
y[0:N] = rand.rand(N)
del(x)
del(y)
if __name__ == '__main__':
test_size()
And the output when profiling with python -m memory_profiler ./testProc.py is:
Line # Mem usage Increment Line Contents
================================================
3 17.699 MiB 0.000 MiB @profile
4 def test_size(N=20e6):
5 17.707 MiB 0.008 MiB x = np.zeros(N, dtype=np.float32)
6 17.711 MiB 0.004 MiB x[0] = 200
7 17.719 MiB 0.008 MiB x[N-1] = 123.555
8 94.031 MiB 76.312 MiB x[0:N] = rand.rand(N)
9 170.332 MiB 76.301 MiB x = np.array(x, copy=False, dtype=np.double, order='C', subok=True)
10 170.340 MiB 0.008 MiB x[0:N] = rand.rand(N)
11 170.344 MiB 0.004 MiB y = np.zeros(N, dtype=np.float32)
12 170.344 MiB 0.000 MiB y[0] = 500.0
13 170.344 MiB 0.000 MiB y[1000] = 123.4
14 246.164 MiB 75.820 MiB y[0:N] = rand.rand(N)
15 93.582 MiB -152.582 MiB del(x)
16 17.285 MiB -76.297 MiB del(y)
As Joe pointed out, the memory profiler's result looks surprising. The profiler is showing memory allocation after each line. The memory footprint enlarges after the lines that fill the arrays, and shrinks when they are deleted. Results from running ps at the shell give a similar result.
Whether the process's deleted memory actually returns to the system stack is another matter.
3 Comments
Joe Kington
Something is going wrong with the profiler. The memory for
x is allocated with you call zeros (should be ~76MB for 20e6 float32's), and when you call np.array(x, copy=False, dtype=np.double) a copy is being made because of the difference in dtype (float32 is not float64).
Frank M
@JoeKington: Yes, the memory is allocated, but that doesn't necessarily mean the process footprint expands immediately. I've expanded to the example to show how the memory footprint behaves. I think this is a reasonable picture, and it shows the effect on the system.
Joe Kington
Yes, which is why the line-by-line memory changes have nothing to do with anything. It's very misleading. For example, your results show no memory change in line 10, despite
rand.rand(N) creating an entire new array of floats before they're assigned to x. (Lines 8 and 10 are exactly equivalent to temp = np.random.rand(N); x[:] = temp.)