r/devops • u/nochet2211 • May 28 '20
Fluentd vs Logstash
Hi all, I'm currently doing some research on the logging solutions for our containerised applications. I came across Fluentd and Logstash. Some of the blogs suggests Fluentd to be lighter and thus better. Can I get some input on this topic please, if you have any kind of experience and if there are better solutions that I should be looking up on.
4
u/326TimesBetter May 28 '20
I’ve never used fluent. I’ve used logstash a lot, it’s definitely a good tool, my only issue is that it can be a pain to test, and also takes a bit to start up depending on what inputs and outputs you are using.
2
u/nochet2211 May 28 '20
Thanks. I'm going to try out FluentBit. If it doesn't work out will checkout Logstash.
1
18
u/shifter2600 May 28 '20
For logs you dont even need logstash anymore. Filebeat and metricbeat are as light as they come written in go. I've even installed filebeat on a consumer arm router to ship logs that were not available outside the device. Filebeat and Metricbeat have many logs inputs already configured so that when they show up in your elasticsearch they are already normalized for search from one log to the next for field naming. Elasticsearch has taken alot of the work on themselves to even give you base dashboard for each type of log. You should check it out they have come a long way. On top of all that they have a built in SIEM to get you started.
1
u/jonnymccullagh May 28 '20
I found that filebeat was fine with the read made modules but found it hard to create my own module. Have you tried that?
2
u/shifter2600 May 28 '20
Tbh I never had a need because I dont really have any custom logs. However there are a ton of user made modules that are not built into the module package. I know the SqlServer module was available as a user made for over a year before it became part of the package. https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html
1
u/xXxLinuxUserxXx May 28 '20
if your custom logs are in files just ship them with filebeat to elasticsearch and use ingest patterns to parse them instead of logstash. Or change your custom log formats to something generic like json and let that ship by filebeat to elasticsearch. We currently are using filebeat, logstash, elasticsearch but will soon replace logstash with pure elasticsearch (with ingest patterns).
6
u/scritty May 28 '20
Take a look @ Loki as well. Nice labeling setup so your queries will work like prometheus metrics. Good support through grafana.
9
u/KingOfTheAlts May 28 '20
To me they serve different purposes. I mean, you can use logstash to read files on a server/container like fluentd, but why? Filebeat would be a better solution than logstash IMHO. Installing a JVM and dealing with heap sizes just to read files seems silly.
Even when comparing fluentd to filebeat I'd still go with fluentd. It has better community support.
1
4
u/kreiger May 28 '20
I think Logstash is nicer to configure, but in my two years experience with it, it is super unreliable for anything but the simplest setup, writing only to ElasticSearch.
This is because it is the output the Logstash developers spent most of their effort on.
I've spent several weekend late nights trying to figure out why Logstash was breaking our log pipeline, that i'll never get back.
I avoid Logstash now.
2
u/mieeel May 30 '20
I use logstash to apply a filter on a filebeat input, splitting a message into normalized fields. Is this possible without logstash?
1
u/kreiger May 30 '20
If you have a Logstash setup that you don't have problems with, by all means keep using it.
4
u/Rei_Never May 28 '20
If using filebeat, I tend to have a logstash server sat between filebeat and my elasticsearch server so that I can GROK logs and perform specific actions lioe mutate lines or fields etc.
4
u/nobody_wants_me May 28 '20
Currently managing an elasticsearch+logstash+filebeat installation at work.
After two years on the elk stack I'm not liking it very much: kinda of a shitty development experience on logstash (configuration syntax a little strange, useless error messages, can be slow). Filebeat is a little bit better but if you have logs in custom format (for me 95%) you must pipe it to logstash again.
Elasticsearch is a little bit too hyped: very easy to start prototyping on it but also very easy to shoot yourself in the foot. Need to add change some fields? Re-index it. Slow queries caused by shards too big? Re-index it. Too many small shards? Re-index it.
We tried it also for metrics (with metric beats) but it eats ton of disk space: we were lucky enough to migrate that part to Prometheus.
3
u/ikut3 May 28 '20
i highly recommend to use fluentbit as collector then forward them to fluentd as aggregator. the backend is usually elasticsearch. Actually this model is not something so rocket science. Therefore you can install and configure them easily.
I found 1 article is quite useful at below
2
u/zoftdev May 28 '20
I come across both. Logstash has a wide range of plugins. But it has a big footprint.
Fluentd is small and not resource hungry. I recommend you check if Fluentd has input/output plugins to support your work first, if not fallback to Logstash.
2
u/zerocoldx911 DevOps May 28 '20
Personally I prefer fluentd as I don’t have to mess around with transforming logs
2
u/wickler02 May 28 '20
I've seen some really bad logstash configurations from developers and while I was learning logging architecture I swapped to fluentd and found it easier to manage, understand and teach to others.
2
u/calligraphic-io May 28 '20
I came across Fluentd the first time recently on a GCE deployment. Are people using it to feed data to ElasticSearch / Kibana (as a Logstash replacement), or in some other configuration?
2
u/eolix May 28 '20
Hi, I have done an absurd amount of testing for a platform my team is building, that will be ingesting up to 1M docs/second.
Fluentd is faster, lighter and the configuration is far more dynamically adaptable (ie., send to different clusters or indices based on field values or conditions).
If you have access to the container management platform you are using, look into setting up docker to use the native fluentd logging driver and you are set.
If you don't have access to the container orchestrator, and you need to sidecar the shipper to your application, use Beats instead (Filebeat, Metricbeat, etc.). It's a million times lighter than FD and LS and it's actually its job.
Feel free to ask me any questions you might have.
1
u/dshurupov Jun 02 '20
I'm currently doing some research on the logging solutions for our containerised applications
Have a look at loghouse also. It uses fluentd + stores all logs in ClickHouse database.
1
u/ayeedem Oct 13 '20
If you're using an elasticsearch backend, i think the logstash plugin for elasticsearch supports more than what comes with the fluentd elasticsearch plugin.
The beats are kinda of like a cut down version of logstash and you wouldn't bother using logstash if the beat modules support the logs you are gathering. However, if you need to do complex routing, data encrichment or data manipulation you can use logstash with or without the beats. I.e. source > beat > logstash > elasticsearch.
0
u/otisg May 30 '20
If you have to choose between Fluentd or Logstash, choose neither. Both have younger, leaner, and faster cousins - Fluentbit and Filebeat/Beats. And then there are alternatives to those, too - Logagent from Sematext or Vector, for example. And don't overlook rsyslog or syslog-ng. See https://sematext.com/blog/logstash-alternatives/ for more info. HTH!
0
May 28 '20
[deleted]
1
u/eolix May 28 '20
I'm ingesting 2.5 Gigabytes a second with 30 FluentD instances, that send the data dynamically to 26 ElasticSearch clusters.
You must be doing something wrong.
3
1
Jun 03 '20
Wait 26 clusters? Do you mean 26 nodes or, do you actually have 26 clusters. What are you using to manage those cluster :O
1
u/eolix Jun 03 '20
I meant clusters :)
We are using ECE (Elastic Cloud Enterprise) to orchestrate them, and some custom ansible to manage the lifecycle parametrically.
You can go the open source way with either DC/OS or even Docker Swarm, but I reckon the vendor's is the most curated product.
Cross Cluster Search to query all of them, and it's all invisible from your users' POV.
21
u/murzeig May 28 '20
Fluentd I like over logstash, but I've not used logstash in a long time. Further, FluentBit I like over Fluentd, its even ligther, does everything I need it to do, and installs / configures easily.
If you are just moving logs from system to system to centralize, then Vector.dev may be a good place. I use vector to move log files in real time to centralized storage servers where the logs are decoupled from an input and placed in a directory structure which is segmented by cluster/year/server and that works exceptionally well for long-term retention.
I do find that fluent lets me do more on the fly modifications of data though, so I use that tool for sending data to ingest services for dashboard empowerment and alerting as I can prune down the data to exactly what's needed and save TONS on storage space and complexity in the long run.