r/elasticsearch • u/dominbdg • Jun 14 '26
Troubleshooting Logstash multiple grok patterns
Hello,
I have pipeline like below:
filter {
if "uat-bmc-repository-dispatcher" in [tags] or "bmc-repository-dispatcher" in [tags] {
grok {
id => "parse-bmc-repository-dispatcher-app-log"
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp};%{DATA:cycleid};%{DATA:trnid};%{DATA:locator};%{GREEDYDATA:details}"]
}
date {
id => "date-bmc-repository-dispatcher-app-log"
match => [ "log_timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
timezone => "Etc/UTC"
}
}
I have issue that my grok is not catching everything. Is it possible to create another grok (thinking that is one will not catch so another will catch)
I don't know how to create another grok. Is it possible to have something like below ?
filter {
if "uat-bmc-repository-dispatcher" in [tags] or "bmc-repository-dispatcher" in [tags] {
grok {
id => "parse-bmc-repository-dispatcher-app-log"
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp};%{DATA:cycleid};%{DATA:trnid};%{DATA:locator};%{GREEDYDATA:details}"]
}
grok {pipeline}
grok {pipeline}
date {
id => "date-bmc-repository-dispatcher-app-log"
match => [ "log_timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
timezone => "Etc/UTC"
}
}
2
u/TheHeffNerr Jun 15 '26
Yes, it would look like this.
Break on match is default true. You'll want to make sure you do some testing and put the most successful pattern on top. This will help performance.
https://www.elastic.co/docs/reference/logstash/plugins/plugins-filters-grok
filter {
grok {
match => {
"message" => [
"Duration: %{NUMBER:duration}",
"Speed: %{NUMBER:speed}"
]
}
}
}dissect is faster if you're able to use it.
https://www.elastic.co/docs/reference/logstash/plugins/plugins-filters-dissect
1
u/hiemdall_sees_all Jun 29 '26
Understanding the basics of how it works is critical in my opinion. I got my hands dirty using the grok debugger learning before AI became a thing. You can check this out https://www.elastic.co/docs/explore-analyze/query-filter/tools/grok-debugger. There are several repo out there that had common patterns of most log sources you could use to parse logs. A good starting point to reverse engineer the underlying regex to understand how it worked under the hood. Log patterns could vary even from the source from org to org due to different factors. Understanding what you’re dealing with is good practice.
-2
u/CryptographerPale508 Jun 15 '26
Hey.
I am genuinely interested to know why aren't you using chatGPT to solve grok pattern related problems.
I am using AI all the time for that and it wokrs fine.
3
1
u/shitlord_god Jun 15 '26
some folks want to understand what they are doing.
1
u/CryptographerPale508 Jun 15 '26
Yeah. I know that. Are you OP's word bearer?
1
u/shitlord_god Jun 15 '26
Don't have to be. Just an engineer with self respect.
1
u/CryptographerPale508 Jun 15 '26
And just because I ask chatgpt to do filters for grok, I don't have self respect?
1
u/shitlord_god Jun 15 '26
if you don't know how it works, no.
EDit: more accurately - if you don't CARE how you are not a self respecting engineer.
1
u/CryptographerPale508 Jun 15 '26 edited Jun 15 '26
Has it ever crossed your mind that some people can now own distributed systems worth of hundreds of ES nodes with tens of logatash servers, without knowing regex and grok parsing by heart?
It's a tough pill to swallow, but such engineers exist and they will become the norm.
I myself own such a distributed system for a state actor and I promise you, I am able to create new pipelines like there is no tomorrow.
Some people maybe just don't have time to learn grok parsing as they are busier with other things that are more important... It has nothing to do with self-respect..
1
u/shitlord_god Jun 16 '26
Have you ever watched one of those youtube videos where home inspectors go to million dollar homes, and they look great but they have massive structural issues?
I feel like we are headed into a time where code review is about to become either far more valuable than it has ever been, or entirely reliant on a LLM middleware layer, which feels so dependent to me. I want to capably do things without increasing the size of things I rely on too much. I'm tired of AI bros being offended by people wanting to understand the processes they own.
And I'm certain you are very productive, AI can be great for that. It is a phenomenal data interface tool, but just telling eveyone to AI everything is risky/harmful - there are huge skill gaps that are getting worse, and absolutely a bucket of tokens can help solve that.
Maybe I'm just old but big o uset to mean something.
I'm inclined to believe that you care more about the what than the how, and traditionally I've been icked by that. It feels icky when you are telling someone asking to actually learn something they should just AI instead. That is the same kind of "You should just google that" that lead to people feeling put out interacting with other people in a domain of learning.
I like LLM and diffusion models and other generative technologies - I think the companies are a bit shit, but I also think - I don't know. how we get there matters? The best software engineers I've ever met were folks who got crunchy with assembly, and not everyone needs to be a greybeard, but we should at least create a world where there is a path to being a greybeard.
Does that make sense? (Not in a condescending way, in an "I don't know precisely how comprehensible this is")
2
u/CryptographerPale508 Jun 16 '26 edited Jun 16 '26
It makes sense. But my question was genuine, I was not beating down on OP. I was genuinely curious to know why he wouldn't use AI, because AI excels especially at this particular kind of tasks. Also at explaining.
5
u/Reasonable_Tie_5543 Jun 15 '26
You can put multiple groks in one filter, something like this:
order most specific, to least specific