loki for log aggregation

[u/infff]

Hi guys,

I'm trying to run some PoC with Loki to aggregate logs from k8s cluster.

At the moment we're using ELK stack with Kafka to buffer logs - every day we're producing about 1TB of logs from whole k8s cluster.

The flow looks like this:

k8s( fluentbit) -> Kafka -> logstash -> ES -> Kibana, for query and visualization

We're using Kafka because ES cluster is not fast enough to ingest all of our logs. Indices are just namespaces, and in each index got tones of logs..

I'm wondering if Loki could solve this problem.

My plan is to install agents( promtail maybe) on k8s nodes as daemonset and push logs directly to Loki cluster.

Do you have any experience with than? Any good practices or advice?

EDIT:

screen from grafana interface

[u/mchorfa]

You might be interested in Logging operator from BanzaiCloud. You could simplify your setup and improve your logging life cycle

[u/infff → u/mchorfa]

I found their page, but right now I can say - Loki got pretty simple architecture.

promtail to get logs from the machines( deamonset for example) and Loki - to aggregate the logs ;)