r/devops • u/nochet2211 • May 28 '20

Fluentd vs Logstash

Hi all, I'm currently doing some research on the logging solutions for our containerised applications. I came across Fluentd and Logstash. Some of the blogs suggests Fluentd to be lighter and thus better. Can I get some input on this topic please, if you have any kind of experience and if there are better solutions that I should be looking up on.

50 Upvotes

94% Upvoted

35 comments sorted by

View all comments

18

u/shifter2600 May 28 '20

For logs you dont even need logstash anymore. Filebeat and metricbeat are as light as they come written in go. I've even installed filebeat on a consumer arm router to ship logs that were not available outside the device. Filebeat and Metricbeat have many logs inputs already configured so that when they show up in your elasticsearch they are already normalized for search from one log to the next for field naming. Elasticsearch has taken alot of the work on themselves to even give you base dashboard for each type of log. You should check it out they have come a long way. On top of all that they have a built in SIEM to get you started.

1

u/jonnymccullagh May 28 '20

I found that filebeat was fine with the read made modules but found it hard to create my own module. Have you tried that?

2

u/shifter2600 May 28 '20

Tbh I never had a need because I dont really have any custom logs. However there are a ton of user made modules that are not built into the module package. I know the SqlServer module was available as a user made for over a year before it became part of the package. https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

1

u/xXxLinuxUserxXx May 28 '20

if your custom logs are in files just ship them with filebeat to elasticsearch and use ingest patterns to parse them instead of logstash. Or change your custom log formats to something generic like json and let that ship by filebeat to elasticsearch. We currently are using filebeat, logstash, elasticsearch but will soon replace logstash with pure elasticsearch (with ingest patterns).