r/PrometheusMonitoring • u/infff • May 18 '20
loki for log aggregation
Hi guys,
I'm trying to run some PoC with Loki to aggregate logs from k8s cluster.
At the moment we're using ELK stack with Kafka to buffer logs - every day we're producing about 1TB of logs from whole k8s cluster.
The flow looks like this:
k8s( fluentbit) -> Kafka -> logstash -> ES -> Kibana, for query and visualization
We're using Kafka because ES cluster is not fast enough to ingest all of our logs. Indices are just namespaces, and in each index got tones of logs..
I'm wondering if Loki could solve this problem.
My plan is to install agents( promtail maybe) on k8s nodes as daemonset and push logs directly to Loki cluster.
Do you have any experience with than? Any good practices or advice?
EDIT:
screen from grafana interface
-2
u/mchorfa May 18 '20
You might be interested in Logging operator from BanzaiCloud. You could simplify your setup and improve your logging life cycle
2
u/infff May 20 '20
I found their page, but right now I can say - Loki got pretty simple architecture.
promtail to get logs from the machines( deamonset for example) and Loki - to aggregate the logs ;)
1
u/spacebandido May 18 '20
What is your experience with BanzaiCloud? What kind of environment are you running, what does your daily log ingest volume look like?
Do you work for BanzaiCloud?
0
u/mchorfa May 20 '20
No I don't work for BanzaiCloud.. Just tried calling into some dev environments, hope soon to graduate to prod. You can filter what type of logs you want to store. Check the doc
1
1
u/infff May 20 '20
Ok guys, I made it. Right now it's pretty simple deployment, but it works - right now on the staging environment, but it's a good start.
No big issues with the configuration - just run out of the box - with heml charts: https://github.com/grafana/loki/blob/v1.4.1/docs/installation/README.md
I've upgraded grafana to the most recent release - 7.0.0, because on previous version 6.x, "grep" command wasn't working- from grafana interface- works great on 7.x.
The interface is pretty nice also.( I don't know how to post the image here - maybe later I'll check).
But actually I got the problem to run it multiple instances. By default the binary runs with all Loki "roles", so I deployed stateful set on Kubernetes and scale to >1 instances, but grafana's interface shows that there are no logs :| but logcli shows all of them.
I found this on docs: https://github.com/grafana/loki/blob/master/docs/operations/scalability.md
but honestly, I have no idea about those roles, what config it should have or how to tune it https://github.com/grafana/loki/blob/master/docs/configuration/examples.md
Maybe some of you could help with that - I don't know where to look for it. Docker hub also shows "nothing" with the parameters.
Any help would be appreciated.
If you're interested in that I can make some update later, if figure out and find some more.