Json parsing Python subprocess

Question

Here's the code:

inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True,  stdout=subprocess.PIPE)
domains = inputDomain.stdout.read().splitlines()

for domain in domains:
   cmd = "whmapi1 domainuserdata domain " + domain
   output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
   jsonS = json.dumps(output.communicate())
   print json.loads(jsonS)['data']

here's there error

root@server [~/testP]# python copie.py Traceback (most recent call last): File "copie.py", line 18, in print json.loads(jsonS)['data'] TypeError: list indices must be integers, not str

and this is an example of the json i need to parse:

{ 
   "data":{ 
      "userdata":{ 
      "phpopenbasedirprotect":1,
      "options":"ExecCGI Includes",
     "ip":"10.0.0.1",
     "hascgi":"1",
     "group":"user",
     "usecanonicalname":"Off",
     "scriptalias":[ 
        { 
           "url":"/cgi-bin/",
           "path":"/home/user/public_html/cgi-bin"
        },
        { 
           "url":"/cgi-bin/",
           "path":"/home/user/public_html/cgi-bin/"
        }
     ],
     "user":"user",
     "ifmodulemodsuphpc":{ 
        "group":"user"
     },
     "owner":"root",
     "documentroot":"/home/user/public_html",
     "userdirprotect":"",
     "serveralias":"parkeddomain.com www.parkeddomain.com www.example.com",
     "port":"80",
     "homedir":"/home/user",
     "ifmoduleconcurrentphpc":{ 

     },
     "customlog":[ 
        { 
           "target":"/usr/local/apache/domlogs/example.com",
           "format":"combined"
        },
        { 
           "target":"/usr/local/apache/domlogs/example.com-bytes_log",
           "format":"\"%{%s}t %I .\\n%{%s}t %O .\""
        }
     ],
     "servername":"example.com",
     "serveradmin":"[email protected]"
  }
}

So i need the user and the domaine, but python always answer that i need a int. Thanks for the help guys.

are you sure jsonS is not a list? in which case you'd have to do json.loads(jsonS)[0]['data'] — Jean-François Fabre
– Jean-François Fabre ♦, Commented Feb 10, 2017 at 14:26
and what the hell with inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True, stdout=subprocess.PIPE)? python can read text files without the need of calling cat !! — Jean-François Fabre
– Jean-François Fabre ♦, Commented Feb 10, 2017 at 14:27
You'r right jf i can change that, but that is not the problem for now.. — marcAndreG
– marcAndreG, Commented Feb 10, 2017 at 14:31

MrMesees · Accepted Answer · 2018-05-12 06:29:59Z

11

I had problems running the above under Python3, so here is a way to achieve what OP is asking in Python3

import subprocess
import json


def getProcessOutput(cmd):
    process = subprocess.Popen(
        cmd,
        shell=True,
        stdout=subprocess.PIPE)
    process.wait()
    data, err = process.communicate()
    if process.returncode is 0:
        return data.decode('utf-8')
    else:
        print("Error:", err)
    return ""


for domain in getProcessOutput("cat /etc/localdomains").splitlines():
    cmd = "whmapi1 domainuserdata domain " + domain
    print(json.loads(getProcessOutput(cmd))['data'])

It outputs Error: None because my machine doesn't have /etc/localdomains but otherwise it seems to work just fine.

answered May 12, 2018 at 6:29

MrMesees

1,63322 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

MrMesees Over a year ago

It should be noted that this is awful code. It seemed to work, but it lacks tests, and couples the success and error into a method, in a way I'm not comfortable with. Something like JavaScript style promises would make me a lot happier for this.

questionto42 Over a year ago

Works for any code, no need for a "data" key in your output of the run CLI command, any command you run can be mapped to the data, err. The data, err = process.communicate() runs the CLI command, the output then is in data, which you then can check as a json object. For example, to get the message content of the chatbot answer from curl, run print(json.loads(data)['choices'][0]['message']['content']). see How can I open a licensed GPT chat like on https://chat.openai.com/ on my own user if I have an API key that is not from my user?.

Jean-François Fabre · Accepted Answer · 2024-04-29 15:53:15Z

8

since your process returns a json string, there's no need to dump it to load it again.

# stdout, stderr
jsonS,_ = output.communicate()

now you have a string, that you can load using json

d = json.loads(jsonS)

now d['data'] yields the info you want

Aside: as I said:

inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True,  stdout=subprocess.PIPE)
domains = inputDomain.stdout.read().splitlines()

could be replaced by native python:

with open("/etc/localdomains") as f: domains = f.read().splitlines()

edited Apr 29, 2024 at 15:53

answered Feb 10, 2017 at 14:37

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

14 Comments

marcAndreG Over a year ago

Hi thanks but i already try tho, this is the error i got when i do this..TypeError: expected string or buffer So its not a string.. its kinda strange

Jean-François Fabre Over a year ago

can you type jsonS,_ = output.communicate(), then print(jsonS)

marcAndreG Over a year ago

just did output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) jsonS,_ = output.communicate() print json.loads(jsonS) --> ValueError: No JSON object could be decoded

Jean-François Fabre Over a year ago

just print(jsonS) don't try to loads.

marcAndreG Over a year ago

ican print it i see the value its like a tuple

|

Vlad · Accepted Answer · 2017-02-10 14:35:09Z

2

communicate() returns a tuple: (stdoutdata, stderrdata). when you loads() it back you get a list, and then trying to index it with 'data' will fail. Example:

import json
import subprocess

cmd = "/bin/date"
output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
jsonS = json.dumps(output.communicate())
print jsonS # ["Fri Feb 10 14:33:42 GMT 2017\n", null]
print json.loads(jsonS)['data'] # TypeError: list indices must be integers, not str

It might be enough to do

jsonS = output.communicate()[0]

answered Feb 10, 2017 at 14:35

Vlad

18.8k4 gold badges44 silver badges72 bronze badges

6 Comments

marcAndreG Over a year ago

it give me back ValueError: No JSON object could be decoded

Vlad Over a year ago

Ok. Stick a print on the jsonS to see whether the data actually is a JSON.

marcAndreG Over a year ago

it says that jsonS is a tuple

Vlad Over a year ago

I meant, after you make the change Jean and I suggested, to take the first element of the return value of communicate().

yeaaaahhhh..hamf hamf Over a year ago

'data' = output.communicate()[0]

|

Collectives™ on Stack Overflow

Json parsing Python subprocess

3 Answers 3

2 Comments

14 Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

14 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related