Apply function returning a 1D array to all elements of numpy array

Question

Say I have an array of d dimensions, and I want to apply a function which returns a 1D array to each element, yielding a d+1-dimensional array. Very much as an embedding lookup works. For instance:

def f(x):
    return np.array([x * i for i in range(6)])

m = np.random.randint(0,10, (2,3,4,5))

g = np.vectorize(f)
h = np.vectorize(f, otypes=[np.ndarray])

I expect n=g(m) to have shape (2,3,4,5,6) just as if I had written 4 for loops, applying f to each element in m.

However, g raises an error: ValueError: setting an array element with a sequence. and h returns an odd array ending like:

...
[array([0, 9, 18, 27, 36, 45, 54, 63, 72]),
array([0, 5, 10, 15, 20, 25, 30, 35, 40]),
array([0, 7, 14, 21, 28, 35, 42, 49, 56]), ...,
array([0, 1, 2, 3, 4, 5, 6, 7, 8]),
array([0, 4, 8, 12, 16, 20, 24, 28, 32]),
array([0, 3, 6, 9, 12, 15, 18, 21, 24])]]]], dtype=object)

which I can't change to what I expect.

I get @hpaulj's comment, I just can't find how to do otherwise.

That's normal. Your f returns an array for a scalar x. Creates a result array of shape matching x. It can only put the array in an object dtype array. — hpaulj
– hpaulj, Commented Jul 26, 2018 at 15:23
I should rephrase: I get why there is an issue, can you think of an alternatives which does not imply for loops? — ted
– ted, Commented Jul 26, 2018 at 15:24
vectorize iterates on elements of x; it just hides that. It's slower than an explicit loop. — hpaulj
– hpaulj, Commented Jul 26, 2018 at 15:26

hpaulj · Accepted Answer · 2018-07-26 17:30:11Z

vectorize has added a signature parameter that does what you want. However it is even slower than the otypes approach. Still for completeness, I'll illustrate:

In [198]: def f(x):
     ...:     return np.array([x * i for i in range(6)])
In [199]: g = np.vectorize(f, signature='()->(6)')

It takes any shape array, and returns a new one with an added size 6 dimension:

In [202]: g(1)
Out[202]: array([0, 1, 2, 3, 4, 5])
In [203]: g(np.array([1,2]))
Out[203]: 
array([[ 0,  1,  2,  3,  4,  5],
       [ 0,  2,  4,  6,  8, 10]])
In [204]: g(np.array([[1],[2]]))
Out[204]: 
array([[[ 0,  1,  2,  3,  4,  5]],

       [[ 0,  2,  4,  6,  8, 10]]])

To do this faster, you have write f in a way that makes use of the compiled numpy methods. There isn't a function that will take a Python function and rewrite it to be faster.

For this f we can take advantage of numpy broadcasting and write:

In [205]: def fn(x):
     ...:     return x[...,None] * np.arange(6)
     ...: 

In [207]: fn(np.array(1))
Out[207]: array([0, 1, 2, 3, 4, 5])
In [208]: fn(np.array([1,2]))
Out[208]: 
array([[ 0,  1,  2,  3,  4,  5],
       [ 0,  2,  4,  6,  8, 10]])
In [209]: fn(np.array([[1],[2]]))
Out[209]: 
array([[[ 0,  1,  2,  3,  4,  5]],

       [[ 0,  2,  4,  6,  8, 10]]])

Here the vectorization was easy. Sometimes it requires some clever thinking, and sometimes it's impossible (especially if the problem is inherently serial in nature). There are tools for producing faster compiled code from iterative solutions, such as numba and cython.

Collectives™ on Stack Overflow

Apply function returning a 1D array to all elements of numpy array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related