sorting numpy structured and record arrays is very slow

Question

it looks like sorting numpy structured and record arrays by a single column is much slower than doing a sort on a similar standalone array:

In [111]: a = np.random.rand(1e4)

In [112]: b = np.random.rand(1e4)

In [113]: rec = np.rec.fromarrays([a,b])

In [114]: timeit rec.argsort(order='f0')
100 loops, best of 3: 18.8 ms per loop

In [115]: timeit a.argsort()
1000 loops, best of 3: 891 µs per loop

There is a marginal improvement using the structured array, but it's not dramatic:

In [120]: struct = np.empty(len(a),dtype=[('a','f8'),('b','f8')])

In [121]: struct['a'] = a

In [122]: struct['b'] = b

In [124]: timeit struct.argsort(order='a')
100 loops, best of 3: 15.8 ms per loop

This indicates that it's potentially faster to create an index array from argsort and then use that to reorder the individual arrays. This is OK except that I expect to be dealing with very large arrays and would like to avoid copying data as much as possible. Is there a more efficient way of doing this that I'm missing?

Jaime · Accepted Answer · 2013-10-30 16:26:50Z

4

What´s slowing you is the use of order, not the fact that you have a record array. If you want to sort by a single field, do it like this:

In [12]: %timeit np.argsort(rec['f0'])
1000 loops, best of 3: 829 us per loop

Once order is used, performance goes south no matter how many fields you want to sort by:

In [16]: %timeit np.argsort(rec, order=['f0'])
10 loops, best of 3: 27.9 ms per loop

In [17]: %timeit np.argsort(rec, order=['f0', 'f1'])
10 loops, best of 3: 28.4 ms per loop

answered Oct 30, 2013 at 16:26

Jaime

67.7k19 gold badges128 silver badges164 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rok Over a year ago

Aha! I figured order did the np.argsort() under the hood, but I guess not?

Rok Over a year ago

but actually, this doesn't solve the issue of copying the data -- it will require me to pass the indices returned by argsort which will result in a copy.

imsc · Accepted Answer · 2014-04-25 21:27:08Z

3

As Jaime have said, you can use argsort to sort the record array.

inds = np.argsort(rec['f0'])

And use take to avoid making a copy

np.take(rec, inds, out=rec)

answered Apr 25, 2014 at 21:27

imsc

7,8807 gold badges49 silver badges70 bronze badges

1 Comment

Jaime Over a year ago

The only reason that works is because np.take makes a copy when you specify an out parameter and leave the mode one in its default 'raise' state, you can look at the source. If you use another mode there will be no copy, but the output will be rubbish, with some values repeated several times and others missing altogether.

Collectives™ on Stack Overflow

sorting numpy structured and record arrays is very slow

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related