Create custom grok pattern to message filed in elasticsearch

Question

I am having a query related to grok processor. For example this is my message filed

{
"message":"agentId:agent003"
}

I want to Grok this and my output should me something like this

{
"message":"agentId:agent003",
"agentId":"agent003"
}

Could some one help me on this how to achieve this? If i am able to do it for one field i can manage for rest of my fields. Thanks in advance.

This is the pipeline i have created in elasticsearch

PUT _ingest/pipeline/dissectpipeline
{
  "description" : "split message content",
  "processors": [
    {
      "dissect": {
      "field": "message",
      "pattern" : "%{apm_application_message.agentId}:%{apm_application_message.agentId}"
   }
    }
  ]
}

Central management added filebeat module other config

- pipeline:
    if: ctx.first_char == '{'
    name: '{< IngestPipeline "dissectpipeline" >}'

There is no error with my filebeat it's working fine but i am unable to find any field like apm_application_message.agentId in index.

How to make sure my pipeline working or not. Also if i am doing something wrong please let me know.

Val · Accepted Answer · 2020-06-17 07:21:24Z

1

Instead of grok I'd suggest using the dissect filter instead with, which is more intuitive and easier to use.

dissect {
  mapping => {
    "message" => "%{?agentId}:%{&agentId}"
  }
}

If you're using Filebeat, there is also the possibility to use the dissect processor:

processors:
  - dissect:
      tokenizer: "%{?agentId}:%{&agentId}"
      field: "message"
      target_prefix: ""

edited Jun 17, 2020 at 7:21

answered Jun 17, 2020 at 6:32

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

sanjay m Over a year ago

If i use this processor in pipeline it is working fine in kibana console. but my intention is to dissect it when shipping the data to elasticsearch using filebeat. Could you please let me know how to use this processor in filebeat.

sanjay m Over a year ago

Note : i am shipping the custom log data of java application from log files using ECS logging for java applications using filebeat

Val Over a year ago

Ok, I thought you were using Logstash, please see my updated answer.

sanjay m Over a year ago

My filebeat.yml is auto generated in the sense i have configured everything in beats central management. In that case where can we add the processor?

Val Over a year ago

You should know, though, that Beats Central Management will be discontinued for a better solution coming up in 7.8.

|

Collectives™ on Stack Overflow

Create custom grok pattern to message filed in elasticsearch

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related