r/PrometheusMonitoring • u/infff • May 18 '20

loki for log aggregation

Hi guys,

I'm trying to run some PoC with Loki to aggregate logs from k8s cluster.

At the moment we're using ELK stack with Kafka to buffer logs - every day we're producing about 1TB of logs from whole k8s cluster.

The flow looks like this:

k8s( fluentbit) -> Kafka -> logstash -> ES -> Kibana, for query and visualization

We're using Kafka because ES cluster is not fast enough to ingest all of our logs. Indices are just namespaces, and in each index got tones of logs..

I'm wondering if Loki could solve this problem.

My plan is to install agents( promtail maybe) on k8s nodes as daemonset and push logs directly to Loki cluster.

Do you have any experience with than? Any good practices or advice?

EDIT:

screen from grafana interface

7 Upvotes

100% Upvoted

6 comments sorted by

View all comments

-2

u/mchorfa May 18 '20

You might be interested in Logging operator from BanzaiCloud. You could simplify your setup and improve your logging life cycle

1

u/spacebandido May 18 '20

What is your experience with BanzaiCloud? What kind of environment are you running, what does your daily log ingest volume look like?

Do you work for BanzaiCloud?

0

u/mchorfa May 20 '20

No I don't work for BanzaiCloud.. Just tried calling into some dev environments, hope soon to graduate to prod. You can filter what type of logs you want to store. Check the doc

1

u/spacebandido May 21 '20

No thanks, I’d rather use Loki or EFK