Python: vectorizing a function call which uses an array of objects

Question

I have an array of objects. I also have a function that requires information from 2 of the objects at a time. I would like to vectorize the call to the function so that it calculates all calls at once, rather than using a loop to go through the necessary pair of objects.

I have gotten this to work if I instead create an array with the necessary data. However this partially defeats the purpose of using objects.

Here is the code. It currently works using the array method and only one line needs to be commented/uncommented in the function to switch to the "object" mode that does not work, but I dearly wish would.

The error I get is: TypeError: only integer arrays with one element can be converted to an index

import numpy as np
import time as time

class ExampleObject():

    def __init__(self, r):
        self.r       = r

def ExampleFunction(x):
    """ WHAT I REALLY WANT """
#    answer = exampleList[x].r - exampleList[indexArray].r 
    """WHAT I AM STUCK WITH """
    answer = coords[x] - exampleList[indexArray].r
    return answer

indexArray = 5   #arbitrary choice of array index
sizeArray = 1000    

exampleList = []
for i in range(sizeArray):
    r = np.random.rand()
    exampleList.append( ExampleObject( r ) )

index_list = np.arange(0,sizeArray,1)
index_list = np.delete(index_list,indexArray)

coords = np.array([h.r for h in exampleList])

answerArray = ExampleFunction(index_list)

The issue is that when I pass the function an array of integers, it doesn't return an array of answers (the vectorization I want) when I use the array (actually, list) of objects. It does work if I use an array (with no objects, just data in each element). But as I have said, this defeats in my mind, the purpose of storing information on objects to begin with. Do I really need to ALSO store the same information in arrays?

I have't digested your code yet, but keep in mind that the quick numpy math only works with numeric dtypes. Object dtypes just contain pointers to objects elsewhere in memory. Math on that kind of array has to iterate, much like it would with a list of the same objects. — hpaulj
– hpaulj, Commented Aug 6, 2018 at 6:18
You get the error because you can index a list with a list - multiple values. You can index an object dtype array with the list. But you can't extract the r attribute from all those objects at once. You have to use the coords like construct to do that. — hpaulj
– hpaulj, Commented Aug 6, 2018 at 6:35

eisbehr · Accepted Answer · 2018-08-06 07:34:58Z

3

I can't comment, sorry for misusing the answer section...

If the data type of a numpy array is python object, the memory of the numpy array is not contiguous. Vectorization of the operation may not improve the performance much if any. Perhaps you might want to try numpy structured array instead.

assume the object has attributes a & b and they are double precision floating point number, then...

import numpy as np

numberOfObjects = 6

myStructuredArray = np.zeros(
    (numberOfObjects,),
    [("a", "f8"), ("b", "f8")],
)

you can initialize individual attributes for say object 0 like this

myStructuredArray["a"][0] = 1.0

or you can initialize individual attributes for all objects like this

myStructuredArray["a"] = [1,2,3,4,5,6]

print(myStructuredArray)

[(1., 0.) (2., 0.) (3., 0.) (4., 0.) (5., 0.) (6., 0.)]

edited Aug 6, 2018 at 7:34

eisbehr

12.5k8 gold badges42 silver badges65 bronze badges

answered Aug 6, 2018 at 6:19

hyl

3691 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

hpaulj Over a year ago

Expand on your comment by creating such a structured array. I've done it, but I want you to get the credit.

hpaulj · Accepted Answer · 2018-08-06 06:54:22Z

numpy.ufunc when given an object dtype array, iterate through the array, and try to apply a cooresponding method to each element.

For example np.abs tries to apply the __abs__ method. Lets add such a method to your class:

In [31]: class ExampleObject():
    ...: 
    ...:     def __init__(self, r):
    ...:         self.r       = r
    ...:     def __abs__(self):
    ...:         return self.r
    ...:

Now create your arrays:

In [32]: indexArray = 5   #arbitrary choice of array index
    ...: sizeArray = 10  
    ...: 
    ...: exampleList = []
    ...: for i in range(sizeArray):
    ...:     r = np.random.rand()
    ...:     exampleList.append( ExampleObject( r ) )
    ...: 
    ...: index_list = np.arange(0,sizeArray,1)
    ...: index_list = np.delete(index_list,indexArray)
    ...: 
    ...: coords = np.array([h.r for h in exampleList])

and make an object dtype array from the list:

In [33]: exampleArr = np.array(exampleList)

In [34]: exampleArr
Out[34]: 
array([<__main__.ExampleObject object at 0x7fbb541eb9b0>,
       <__main__.ExampleObject object at 0x7fbb541eba90>,
       <__main__.ExampleObject object at 0x7fbb541eb3c8>,
       <__main__.ExampleObject object at 0x7fbb541eb978>,
       <__main__.ExampleObject object at 0x7fbb541eb208>,
       <__main__.ExampleObject object at 0x7fbb541eb128>,
       <__main__.ExampleObject object at 0x7fbb541eb198>,
       <__main__.ExampleObject object at 0x7fbb541eb358>,
       <__main__.ExampleObject object at 0x7fbb541eb4e0>,
       <__main__.ExampleObject object at 0x7fbb541eb048>], dtype=object)

Now we can get an array of the r values by calling the np.abs function:

In [35]: np.abs(exampleArr)
Out[35]: 
array([0.28411876298913485, 0.5807617042932764, 0.30566195995294954,
       0.39564156171554554, 0.28951905026871105, 0.5500945908978057,
       0.40908712567465855, 0.6469497088949425, 0.7480045751535003,
       0.710425181488751], dtype=object)

It also works with indexed elements of the array:

In [36]: np.abs(exampleArr[:3])
Out[36]: 
array([0.28411876298913485, 0.5807617042932764, 0.30566195995294954],
      dtype=object)

This is convenient, but I can't promise speed. In other tests I found that iteration over object dtypes is faster than iteration (in Python) over numeric array elements, but slower than list iteration.

In [37]: timeit np.abs(exampleArr)
3.61 µs ± 131 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [38]: timeit [h.r for h in exampleList]
985 ns ± 31.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [39]: timeit np.array([h.r for h in exampleList])
3.55 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Using this technique you have converted the list of objects to a numpy array of objects. Is there any reason for me to keep my list of objects? I made a list of objects out of naive ignorance... if each time I really use it I need to convert it to an array, should I just make it a numpy array to start with and save some time in converting each loop? In the real program this gets looped over 10+million times :)
That doesn't have a general answer. It depends on how else you intend to construct an object array. List append is faster than array concatenates, and competitive with iteration through a preallocated array.

Collectives™ on Stack Overflow

Python: vectorizing a function call which uses an array of objects

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related