ElasticSearch Grok Pattern issue for Custom Log String

Question

I am trying to create a complete GROK pattern on Elasticsearch for the following custom JSON based log:

------------------------DEBUG----------------------------
Date   : 2019-12-26 12:18:21,498 
METHOD NAME:   xyz
{
  "methodName": "SMS_POOL_IN",
  "Tran_Type": "Response",
  "URL": "xyz.abcL",
  "ApiResult": "Success",
  "Date": "2019/12/26 12:18:21",
  "ErrorCode": "00",
  "ErrorReason": "Success",
  "Msisdn": "9999999",
  "CNIC": "99999999",
  "RequestID": "1111",
  "SR_TranID": "2222",
  "Channel": "abc"
}

but when ever I parse this, i get only timestamp from the grok.

I am using grok debugger to test this. Whenever i use greedydata, i get only first json parameter, rest gets ignored, am i missing something here? how can i make a grok from these logs? Any helping hand would be appreciated

I have created below grok

%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}

and getting following result with this.

{
  "GREEDYDATA": [
    [
      "------------------------DEBUG----------------------------",
      "Date   : 2019-12-26 12:18:21,498 ",
      "METHOD NAME:   xyz",
      "{",
      ""methodName": "SMS_POOL_IN",",
      ""Tran_Type": "Response",",
      ""URL": "xyz.abcL",",
      ""ApiResult": "Success",",
      ""Date": "2019/12/26 12:18:21",",
      ""ErrorCode": "00",",
      ""ErrorReason": "Success",",
      ""Msisdn": "9999999",",
      ""CNIC": "99999999",",
      ""RequestID": "1111",",
      ""SR_TranID": "2222",",
      ""Channel": "abc"",
      "} ",
      "",
      ""
    ]
  ],
  "SPACE": [
    [
      "\n",
      "\n",
      "\n",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n",
      "",
      ""
    ]
  ]
}

I need to show all these json tags segregated as i need them to populated in seperate tags in ELK

"how can i make a grok from these logs?". Please let us know what the desired result of the grok pattern should look like! And also, please post what you have already implemented. — apt-get_install_skill
– apt-get_install_skill, Commented Dec 26, 2019 at 22:21
@apt-get_install_skill i have edited the question, I have shared complete details — Hassaan Rabbani
– Hassaan Rabbani, Commented Dec 27, 2019 at 6:59

Hassaan Rabbani · Accepted Answer · 2019-12-28 15:33:34Z

I have created the grok myself, the only issue was with the syntax i was trying to build the grok. Below is the correct grok syntax to read above

%{TIMESTAMP_ISO8601:date_time}\s*%{GREEDYDATA:Method}\n%{GREEDYDATA:Bracket}\s*\"methodName\"\:\s\"%{DATA:methodName}\s*\"Tran_Type\"\:\s\"%{DATA:Tran_Type}\s*\"URL\"\:\s\"%{DATA:URL}\s*\"ApiResult\"\:\s\"%{DATA:ApiResult}\s*\"Date\"\:\s\"%{DATA:Date}\s*\"ErrorCode\"\:\s\"%{DATA:ErrorCode}\s*\"ErrorReason\"\:\s\"%{DATA:ErrorReason}\s*\"Msisdn\"\:\s\"%{DATA:Msisdn}\s*\"CNIC\"\:\s\"%{DATA:CNIC}\s*\"RequestID\"\:\s\"%{DATA:RequestID}\s*\"SR_TranID\"\:\s\"%{DATA:SR_TranID}\s*\"Channel\"\:\s\"%{DATA:Channel}\s

First i picked up time stamp, then i picked up everything which is outside my json string in GREEDYDATA, and then i segregated json tags with DATA keyword.

result for above is

{
  "date_time": [
    [
      "2019-12-26 12:18:21,498"
    ]
  ],
  "YEAR": [
    [
      "2019"
    ]
  ],
  "MONTHNUM": [
    [
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      "26"
    ]
  ],
  "HOUR": [
    [
      "12",
      null
    ]
  ],
  "MINUTE": [
    [
      "18",
      null
    ]
  ],
  "SECOND": [
    [
      "21,498"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "Method": [
    [
      "METHOD NAME:   xyz"
    ]
  ],
  "Bracket": [
    [
      "{"
    ]
  ],
  "methodName": [
    [
      "SMS_POOL_IN","
    ]
  ],
  "Tran_Type": [
    [
      "Response","
    ]
  ],
  "URL": [
    [
      "xyz.abcL","
    ]
  ],
  "ApiResult": [
    [
      "Success","
    ]
  ],
  "Date": [
    [
      "2019/12/26 12:18:21","
    ]
  ],
  "ErrorCode": [
    [
      "00","
    ]
  ],
  "ErrorReason": [
    [
      "Success","
    ]
  ],
  "Msisdn": [
    [
      "9999999","
    ]
  ],
  "CNIC": [
    [
      "99999999","
    ]
  ],
  "RequestID": [
    [
      "1111","
    ]
  ],
  "SR_TranID": [
    [
      "2222","
    ]
  ],
  "Channel": [
    [
      "abc""
    ]
  ]
}

JBone · Accepted Answer · 2019-12-28 15:00:48Z

I assume you wanted to separate out first 3 lines into 3 separate fields and rest of the JSON string object into another field.

I copied your input text from here so every line ended with \n. So this was my pattern match element.

Let me know if the output needed more parsing etc.

my pipeline configuration that has grok pattern to parse the input

input {                                                                                                                            
           http {

           }

   }

  filter {

          grok {
                  match  => { "message" => "(?<debug-string>[^\n]+)\n(?<date-string>[^\n]+)\n(?<method-name>[^\n]+)\n%{GREEDYDATA:RestOfIt}" }
          }
         mutate {
                 remove_field => ["headers", "host", "@timestamp", "@version"]
         }

  }

  output {
          stdout {

          }
}

The OUTPUT

{ "message" => "------------------------DEBUG----------------------------\nDate : 2019-12-26 12:18:21,498 \nMETHOD NAME: xyz\n{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\",\n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12/26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\",\n \"Channel\": \"abc\"\n} ",

"date-string" => "Date : 2019-12-26 12:18:21,498 ",

"method-name" => "METHOD NAME: xyz",

"RestOfIt" => "{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\",\n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12/26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\",\n \"Channel\": \"abc\"\n} ",

"debug-string" => "------------------------DEBUG----------------------------" }

I have posted the result in my answer which i was looking for :)

Collectives™ on Stack Overflow

ElasticSearch Grok Pattern issue for Custom Log String

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related