Numpy/Python filter multiple columns at once

Question

In my text file I have a total of 14 columns. I would like to filter all of them.

data = numpy.genfromtxt("Prob_1.txt",delimiter=" ")

idx = numpy.where(numpy.logical_and(data[:,0] > 2,data[:,1] > 0,data[:,2] > 0, (data[:,10] >= 9) & (data[:,10] <= 15)))
xx = data[idx]

However, I keep getting an value error:

ValueError: invalid number of arguments

This is how the text file looks like.

nan -inf nan nan 1.68268949214 nan 0.0 0.0 1111089600.0 0.68 -2.89 -0.484544695071 1.37812459526 nan
inf -0.875465538612 -inf inf 1.0 1.0 0.5 1.0 1111176000.0 -0.75 0.68 0.0419076355712 -0.41951910247 inf
-0.069227237319 -1.16491504553 -1.18905985655 -0.048951048951 1.43629713835 0.5 0.333333333333 2.0 1111435200.0 1.51 -0.75 -0.478549240198 2.04564432143 -0.749613618659
0.890689464455 -1.25399928255 -2.15522547913 0.514239802067 1.0 0.666666666667 0.25 3.0 1111694400.0 0.37 -0.31 0.0513083632632 -0.192287053268 -0.636520174795
1.11760871074 -1.56075998095 -1.64682922027 0.558804355372 0.654720846019 0.75 0.2 4.0 1112126400.0 -0.12 -0.8 -0.0936808039516 -0.317691281678 -0.53036244861
0.99496493299 -1.56135997804 -1.78538243457 0.444961845079 1.0 0.6 0.166666666667 5.0 1112212800.0 0.06 -0.12 -0.5207444084 0.713761033114 -0.56062270571
1.01897862854 -1.48889020147 -2.40488526577 0.415996283122 0.705456986111 0.666666666667 0.285714285714 6.0 1112299200.0 -0.38 0.06 1.19883541702 0.370549602489 -0.531298287889
0.750060660183 -1.62182088145 -0.19362324657 0.283496282151 1.0 0.571428571429 0.25 7.0 1112385600.0 -1.5 -0.38 -0.194601912175 -0.316796973518 -0.583207744721
-0.0537383386956 -2.05994059479 -0.32407557587 -0.0189993718507 1.26111731964 0.5 0.222222222222 8.0 1112644800.0 -2.36 -1.5 -1.01154851218 0.246594558677 -0.862008726586
-0.75997327926 -2.51345571483 -0.861297350024 -0.25332442642 1.47291074313 0.444444444444 0.2 9.0 1112731200.0 -0.46 -2.36 0.230061349693 0.44614883473 -1.17658513717
-0.899104054379 -2.63627687381 -0.711913333475 -0.284321666533 1.63428770372 0.4 0.181818181818 10.0 1112817600.0 1.44 -0.46 0.348106639497 -0.144302242244 -1.12746763278
-0.410942273785 -2.62695984763 -0.499466998557 -0.123903757513 1.43629713835 0.454545454545 0.166666666667 11.0 1112990400.0 -0.84 -0.13 0.700954311291 0.360814154335 -1.14525640573
-0.629087204679 -2.87224486026 -0.107489621345 -0.181601833483 1.59461944354 0.416666666667 0.153846153846 12.0 1113249600.0 -0.92 -0.84 -0.981790719141 -0.342881646655 -1.13079043244
-0.860568951041 -3.13843058565 -0.547051163868 -0.238678883004 1.7149505926 0.384615384615 0.142857142857 13.0 1113336000.0 1.66 -0.92 -0.169075999662 0.561054554946 -1.12007576947
-0.381268853256 -3.51095009374 -0.62413526043 -0.101898387224 1.56142197392 0.428571428571 0.133333333333 14.0 1113508800.0 -0.67 -2.11 -1.06275303644 -0.10777075881 -1.15580876076

This will need to be done in Numpy/Python; Pandas does not work on my computer.

How can I fix these issues?

logical_and only works for two arrays at a time. Does this answer for logical_or applied to logical_and solve your problem? Those two functions have the same api so I'd say this is more or less a duplicate. — jotasi
– jotasi, Commented Feb 23, 2017 at 8:47
Possible duplicate of Numpy `logical_or` for more than two arguments — jotasi
– jotasi, Commented Feb 23, 2017 at 8:48
Why are you using both logical_and and &? You're aware they mean the same thing, right? — Eric
– Eric, Commented Feb 23, 2017 at 10:08

Amir Ziai · Accepted Answer · 2017-02-23 08:54:51Z

5

numpy.logical_and only takes 2 arguments. See the API reference here.

You can chain your conditions and then apply it to your data for filtering:

mask = (data[:, 0] > 2) & (data[:, 1] > 0) & (data[:,2] > 0)
data[mask]

answered Feb 23, 2017 at 8:54

Amir Ziai

1481 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

aslavkin · Accepted Answer · 2017-02-23 08:51:50Z

1

Your problem with & operator.Just separate in to logical arrays.

a = numpy.logical_and(data[:,0] > 2,data[:,1] > 0,data[:,2] > 0)
b = numpy.logical_and(data[:,10] >= 9,data[:,10] <= 15)

and and between them.

answered Feb 23, 2017 at 8:51

aslavkin

2241 silver badge7 bronze badges

2 Comments

ben olsen Over a year ago

so when I separate the logical arrays a and b how do I combine them after it has been filtered?

aslavkin Over a year ago

numpy.logical_and(a,b) but I got all False values in result array with you input. Check you conditions.

Daniel F · Accepted Answer · 2017-02-23 13:41:36Z

0

You can keep it as one line, but you need to reduce by numpy.logical_and, since it normally only takes 2 arguments.

dx = numpy.where(reduce(numpy.logical_and, [data[:, 0] > 2, data[:, 1] > 0,data[:, 2] > 0, data[:, 10] >= 9, data[:, 10] <= 15]))

answered Feb 23, 2017 at 13:41

Daniel F

14.5k2 gold badges34 silver badges59 bronze badges

Collectives™ on Stack Overflow

Numpy/Python filter multiple columns at once

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related