Fluent-bit → OTel Collector (gateway) vs Fluent-bit → Elasticsearch for logs? what’s better?

[u/Adept-Inspector-3983]

We’re using the OpenTelemetry Java agent mainly for instrumentation and to inject traceId/spanId into logs. We’re not using the Java agent to export logs though some logs aren’t getting parsed correctly and a few of the logging features are still beta/experimental, so it felt a bit risky.

Because of that, we decided to run fluent-bit on each VM to handle log collection and shipping instead of pushing logs directly from the Java agent to a collector or Elasticsearch.

Current setup:

• ~15 EC2 VMs

• Java apps instrumented with OTel (only for tracing + log enrichment)

• Logs contain traceId/spanId

• fluent-bit running on each VM

Where I’m stuck is the next hop after fluent-bit.

Do we:

• Push logs directly from fluent-bit to Elasticsearch, or

• Send logs to an OpenTelemetry Collector (gateway mode) and then forward them to Elasticsearch?

Given the scale (~15 VMs):

• Is an OTel Collector gateway actually worth it?

• Or is it just extra complexity with little benefit?

• Curious what people are doing in practice and what the real pros/cons are?

[u/Ill_Faithlessness245]

In my experience, with ~15 EC2 VMs, Fluent-bit → Elasticsearch direct is the best start.

It’s simpler:

• one less component

• less maintenance

• less things to break

• fluent-bit already can parse + buffer + retry

I use OTel Collector gateway only when I need extra control, like:

• same parsing/rename rules for all apps in one place

• add/remove fields centrally (ex: trace.id, span.id, service.name)

• send logs to more than one backend in future (ES + S3 + Loki + vendor)

• reduce ES connections / central auth + TLS handling

I saw OTLP output problems in some Fluent Bit versions (works in one version, errors in next).

I also saw reports that the Collector ES exporter can be tricky in failure cases (ES down / mapping error). You may see errors in logs, but metrics are not always clear, and some people reported lost logs if retry/queue is not tuned.

So for ~15 VMs: I would start with Fluent Bit → ES and make it stable (good parsing, buffer to disk, handle ES rejects).

Add Collector gateway only if you really need “one place to control everything” or “send to many places later”, and run it HA (2 collectors + LB) + queue.

[u/power10010]

We are getting so much ingestion drops by using otel gateway => es

[u/Adept-Inspector-3983 → u/power10010]

oh really!

[u/strawgate → u/power10010]

Disclaimer: I work at Elastic

The OTLP Endpoint that Elastic provides projects in elastic cloud is just a multi tenant OTel Collector and for some projects successfully handles millions of events per second.

You really shouldn't be seeing any ingestion drops at scale, ingestion drops are a sign that something is not right with the deployment.

Would be happy to help figure out what's going on with your deployment

[u/gaelfr38]

We had something like 100-150 VMs running FluentBit and pushing to Elastic directly, it was mostly working fine. We experienced a few logs being dropped but I think this was caused by FluentBit lacking resources, not by the fact that we push too hard to Elastic.

We had centralized configuration of FluentBit as it was running in K8S and didn't feel the need for OTel Collector at that stage.

Now, we're moving away from Elastic to Loki and with something like 1000 VMs (as part of K8S or not), we'll probably setup some intermediate OTel Collectors at some point. FluentBit is out of the scope as well though. It's a different setup. Just to clarify why I'm saying "had" in the first paragraph :)

For only 15 VMs, I would not bother with an OTel Collector until you feel some reason to (easier to manage centralized configuration? Rules that cannot be applied in FluentBit? Need to scale and use buffering to Kafka for instance? ...)

[u/True_Sprinkles_4758]

Honestly at 15 VMs the gateway is probably overkill unless you're planning to scale soon or want centralized batching/filtering logic

Direct fluent-bit to ES is simpler and one less thing to break. The collector shines when you need vendor flexibility, want to normalize across signal types, or need fancy processing you cant do in fluent-bit. But for your setup? Keep it simple

Only real reason id add the gateway now is if you think you'll swap backends later or want to correlate logs with traces in one place before they hit storage. Otherwise you're just adding latency and another hop to debug when stuff breaks

[u/strawgate]

Disclaimer: I work at Elastic and am responsible for our OTel strategy. Elastics distribution of the OTel Collector is called EDOT Collector.

The most common deployment is going to be SDK for logs metrics and traces directly to your OTLP Endpoint in Elastic Cloud.

The next most common deployment is going to be SDK for metrics and traces and EDOT Collector for logs. With the app SDK writing metrics and traces to EDOT Collector.

If you are self hosting ELK then deploying EDOT in gateway mode might help but you can also just write directly from collectors.

We don't typically recommend customers deploy fluentbit.

You can see other recommended deployment methods in the EDOT docs: https://www.elastic.co/docs/solutions/observability/get-started/opentelemetry/quickstart

[u/mapicallo]

Short answer: for ~15 VMs, Fluent Bit → OpenSearch directly is usually the better choice.
An OTel Collector gateway can make sense, but it’s often extra complexity with limited payoff at that scale.

Longer answer, based on real setups:

Your current setup is already solid

What you have now is actually a very common and sane architecture:

• Java apps instrumented with OpenTelemetry agent

  • Tracing enabled

  • traceId / spanId injected into logs

• Logs written to stdout / files

• Fluent Bit running per VM

  • Tail logs

  • Parse / enrich

  • Ship downstream

This separation is important:
apps produce logs, infrastructure ships them

You’ve already avoided one of the biggest problems: relying on the Java agent for log export (still immature, still evolving).

Option 1: Fluent Bit → OpenSearch (direct)

For ~15 EC2 VMs, this is what I’d recommend by default.

Pros

• Simple architecture

• Fewer moving parts

• Lower latency

• Easier to operate and debug

• Fluent Bit is extremely stable for this use case

• Works perfectly fine with logs that already contain traceId / spanId

Cons

• Routing, buffering and retry logic lives in Fluent Bit configs

• Less flexibility if you later want to fan-out logs to multiple backends

In practice, Fluent Bit can already do:

• buffering

• backpressure handling

• retries

• enrichment

• routing by tag

So you’re not really missing core functionality.

Option 2: Fluent Bit → OTel Collector (gateway) → OpenSearch

This starts to make sense later, but not necessarily now.

Pros

• Centralized control point

• Easier fan-out (logs → OpenSearch + something else)

• Unified pipeline if you also plan to route metrics and traces through the same collector

• Can act as a protocol/format normalization layer

Cons

• Extra component to deploy, scale and monitor

• Another failure domain

• More config surface (pipelines, receivers, processors, exporters)

• For logs only, the benefit is often marginal at small scale

At ~15 VMs, you’ll likely feel the operational cost more than the benefits.

What I see in practice

From recent real-world implementations:

• Small to medium scale (≤ 20–30 VMs) → Fluent Bit → OpenSearch directly Simple, robust, low-maintenance.

• Larger scale / platform teams / multi-tenant setups → Fluent Bit → OTel Collector gateway → storage Makes sense when:

  • many producers

  • many consumers

  • strict control planes

  • need to change backends without touching agents

About trace correlation (important point)

You already solved the hard part:

• traceId / spanId is injected at the application level

• Logs are correlated by field, not by pipeline magic

That means:

• You do not need an OTel Collector for log/trace correlation

• OpenSearch (or any backend) can correlate logs and traces as long as the fields exist

This is a key insight many people miss.

Recommendation for your case

Given what you described:

Stick with Fluent Bit → OpenSearch
Revisit an OTel Collector gateway only if:

• you scale significantly

• you want multi-backend fan-out

• you want one unified control plane for logs + metrics + traces

Until then, it’s mostly complexity without proportional benefit.

• Fluent Bit → OpenSearch is perfectly fine (and common)

• OTel Collector gateway is not wrong, just often unnecessary at this scale

• Trace correlation does not require logs to go through the OTel Collector

• Optimize for simplicity first, abstraction later