Custom GROK filter - Logstash -> Elasticsearch

Question

I have a log which is getting captured and sent to logstash, the format of the log is

22304999    5   400.OUTPUT_SERVICE.510  submit  The limit has been exceeded. Please use a different option. 2.54.44.221 /api/output/v3/contract/:PCID/order /api/output/v3/contract/:pcid/order https://www.example.org/output/ PUT 400 2017-09-28T15:50:57.843176Z

I am trying to create a custom grok filter to add the header fields before it gets sent to elasticsearch.

My aim is something like this,

 SessionID   => "22304999"
 HitNumber   => "5"
 FactValue   => "400.OUTPUT_SERVICE.510"
 DimValue1   => "submit"
 ErrMessage  => "The limit has been exceeded. Please use a different option."
 IP          => "2.54.44.221"
 TLT_URL     => "/api/output/v3/contract/:PCID/order"
 URL         => "/api/output/v3/contract/:pcid/order"
 Refferer    => "https://www.example.org/output/"
 Method      => "PUT"
 StatsCode   => "400"
 ReqTime     => "2017-09-28T15:50:57.843176Z"

I am new to this so only trying to understand how I apply and test this, for example I would start with an empty filter,

filter {
   grok {
     match => { "message" => "" }
   }
 }

My first question is match => { "message" => "" } is message just a log line? What defines 'message'?

My log and the fields I want are separated by a Tab, after each Tab its a new field, would this make what I am trying to achieve easier, rather than looking for a pattern can I just look for the next Tab?

Failing this, could someone provide an example for one of my fields, from that I should be able to complete the rest.

Haha what a guy! I wanted somethings explaining but that page also does that! (?<SessionID> is how I define a group? Excellent!! thank you — user3236169
– user3236169, Commented Jan 30, 2018 at 13:17

Srdjan M. · Accepted Answer · 2018-01-30 13:35:03Z

2

Regex: (?<SessionID>\S+)\s+(?<HitNumber>\S+)\s+(?<FactValue>\S+)\s+(?<DimValue1>\S+)\s+(?<ErrMessage>.+)\s+(?<IP>(?:\d{1,3}\.){3}\d{1,3})\s+(?<TLT_URL>\S+)\s+(?<URL>\S+)\s+(?<Refferer>\S+)\s+(?<Method>\S+)\s+(?<StatsCode>\S+)\s+(?<ReqTime>\S+)

Details:

(?<>) Named Capture Group
\S matches any non-whitespace character
\d Matches a digit, {n,m} Matches between n and m times
+ Matches between one and unlimited times

Regex demo

Output:

{
  "SessionID": [
    [
      "22304999"
    ]
  ],
  "HitNumber": [
    [
      "5"
    ]
  ],
  "FactValue": [
    [
      "400.OUTPUT_SERVICE.510"
    ]
  ],
  "DimValue1": [
    [
      "submit"
    ]
  ],
  "ErrMessage": [
    [
      "The limit has been exceeded. Please use a different option."
    ]
  ],
  "IP": [
    [
      "2.54.44.221"
    ]
  ],
  "TLT_URL": [
    [
      "/api/output/v3/contract/:PCID/order"
    ]
  ],
  "URL": [
    [
      "/api/output/v3/contract/:pcid/order"
    ]
  ],
  "Refferer": [
    [
      "https://www.example.org/output/"
    ]
  ],
  "Method": [
    [
      "PUT"
    ]
  ],
  "StatsCode": [
    [
      "400"
    ]
  ],
  "ReqTime": [
    [
      "2017-09-28T15:50:57.843176Z"
    ]
  ]
}

edited Jan 30, 2018 at 13:35

answered Jan 30, 2018 at 13:19

Srdjan M.

3,4253 gold badges17 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3236169 Over a year ago

Thank you, could you explain how this would fit into filter { grok { match => { "message" => "" } } }

Wiktor Stribiżew Over a year ago

Shorten all [^\s] to \S

MrSimple · Accepted Answer · 2018-01-30 13:30:08Z

2

If you are testing a solution, you can always use this site:

http://grokconstructor.appspot.com/do/match

I made this grok pattern for your problem:

%{INT:SessionID}\s*%{INT:HitNumber}\s*%{NOTSPACE:FaceValue}\s*%{GREEDYDATA:ErrMessage}\s*%{IP:IP}\s*%{NOTSPACE:TLT_URL}\s*%{NOTSPACE:URL}\s*%{NOTSPACE:Referer}\s*%{NOTSPACE:Method}\s*%{INT:StatsCode}\s*%{TIMESTAMP_ISO8601:ReqTime}

answered Jan 30, 2018 at 13:30

MrSimple

5994 silver badges14 bronze badges

1 Comment

user3236169 Over a year ago

Ahh this is much more like what I was expecting so see, I am right in thinking to allow this for logstash I can do filter { grok { match => { "message" => "GROK PATTERN HERE" } } }?

Collectives™ on Stack Overflow

Custom GROK filter - Logstash -> Elasticsearch

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related