We have a syslog-ng server that get Mongodb logs that contain JSON in the log, The sample log is below:
2016-10-17 19:54:12 f:local1.p:info h:10.133.126.79 prog:sharmongo-log m:sharmongo-log 2016-10-17T19:54:04.943+0330 I COMMAND [conn5573] command CLM.BillingAccount command: count { count: "BillingAccount", query: { $or: [ { billingAccount.customerCode: "C8088810719" } ] } } planSummary: COUNT_SCAN { billingAccount.customerCode: 1.0 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:122 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 216ms
We need to insert the incomming logs to the database, first of all I need to parse the log to CSV, I want to write a python script to change the incomming logs to CSV. as you can see a part of the log is JSON (as described in MongoDB documentation ant this girthub link it is in JSON format). the JSON portion of logs are like this:
command: count {
count: "BillingAccount",
query: {
$or: [{
billingAccount.customerCode: "C8088810719"
}]
}
}
planSummary: COUNT_SCAN {
billingAccount.customerCode: 1.0
}
keyUpdates: 0 writeConflicts: 0 numYields: 0 reslen: 122 locks: {
Global: {
acquireCount: {
r: 2
}
},
Database: {
acquireCount: {
r: 1
}
},
Collection: {
acquireCount: {
r: 1
}
}
}
protocol: op_query 216 ms
but I see a lot of errors in it that when I want to parse it with a python json library I get an error. I need to change it to a form that I doesn't give me an error anymore, first I quoted every key as they were not. but again I got the following errors:
besides these errors' one major problem is that I got more than one JSON in log that it caused me more errors , I need to know If there is a python library to help me to parse this JSON. What do you suggest to solve this problem?


{}so that it becomes a json object.