Fastest way to parse JSON strings into numpy arrays

Question

I have huge json objects containing 2D lists of coordinates that I need to transform into numpy arrays for processing.

However using json.loads followed with np.array() is too slow.

Is there a way to increase the speed of creation of numpy arrays from json?

import json
import numpy as np

json_input = '{"rings" : [[[-8081441.0, 5685214.0], [-8081446.0, 5685216.0], [-8081442.0, 5685219.0], [-8081440.0, 5685211.0], [-8081441.0, 5685214.0]]]}'

dict = json.loads(json_input)
numpy_2d_arrays = [np.array(ring) for ring in dict["rings"]]

I would take any solution whatsoever!

I'm getting a json decoder error: ` File "/usr/lib/python3.5,... json.decoder.JSONDecodeError: Expecting value: line 1 column 14 (char 13)` — hpaulj
– hpaulj, Commented Dec 9, 2016 at 23:15
Other parsers are eval and ast.literal_eval (safer). On this small sample json.loads is noticeably faster. The np.array part takes even less time. — hpaulj
– hpaulj, Commented Dec 10, 2016 at 1:32

Community · Accepted Answer · 2017-05-23 12:24:53Z

6

The simplest answer would just be:

numpy_2d_arrays = np.array(dict["rings"])

As this avoids explicitly looping over your array in python you would probably see a modest speedup. If you have control over the creation of json_input it would be better to write out as a serial array. A version is here.

edited May 23, 2017 at 12:24

CommunityBot

11 silver badge

answered Dec 9, 2016 at 21:19

Daniel

19.6k7 gold badges64 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Laurent LAPORTE · Accepted Answer · 2016-12-11 15:58:11Z

4

Since JSON syntax is really near to Python syntax, I suggest you to use ast.literal_eval. It may be faster…

import ast
import numpy as np

json_input = """{"rings" : [[[-8081441.0, 5685214.0],
                             [-8081446.0, 5685216.0],
                             [-8081442.0, 5685219.0],
                             [-8081440.0, 5685211.0],
                             [-8081441.0, 5685214.0]]]}"""

rings = ast.literal_eval(json_input)
numpy_2d_arrays = [np.array(ring) for ring in rings["rings"]]

Give it a try. And tell us.

answered Dec 11, 2016 at 15:58

Laurent LAPORTE

23.2k7 gold badges64 silver badges111 bronze badges

1 Comment

Below the Radar Over a year ago

Thank you but ast.literal_eval is slower than json.loads() with my data

Gribouillis · Accepted Answer · 2016-12-09 21:29:17Z

3

For this specific data, you could try this

import numpy as np

json_input = '{"rings" : [[(-8081441.0, 5685214.0), (-8081446.0, 5685216.0), (-8081442.0, 5685219.0), (-8081440.0, 5685211.0), (-8081441.0, 5685214.0)]]}'
i = json_input.find('[')
L = eval(json_input[i+1:-2])
print(np.array(L))

answered Dec 9, 2016 at 21:29

Gribouillis

2,2401 gold badge11 silver badges16 bronze badges

2 Comments

ei-grad Over a year ago

using eval is unsafe

Gribouillis Over a year ago

@ei-grad It depends on the context. Python has always been made for consenting adults!

Collectives™ on Stack Overflow

Fastest way to parse JSON strings into numpy arrays

3 Answers 3

Comments

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related