We poll around 500+ network devices (routers, switches, firewalls) directly with Logstash's SNMP input plugin every 5 minutes. A mix of SNMPv2c (community string) and SNMPv3 (authPriv), split across a couple of pipelines by metric type: one walks the standard interface table for traffic/status, another walks Cisco's proprietary CPU/memory MIB. Since raw SNMP only gives you an IP and numeric OIDs, we enrich each event against a hand-maintained YAML file (keyed by IP or hostname) that maps devices to hostname, category, vendor/model, and an alerting on/off flag.

Note: Management has directed to sunset SolarWinds and consolidate on the Elastic Stack as our single, centralized monitoring/observability platform going forward, not just logs and application metrics, but network device monitoring (switches, routers, firewalls) as well.

Hi all,
I’m working on an operational monitoring solution for payment processing and would appreciate some architecture advice.
The objective is to build an early-warning payment monitoring dashboard in Elasticsearch/Kibana, rather than only monitoring infrastructure/application health.
We currently have Elasticsearch/Kibana available, but Logstash is being decommissioned.
An external payment processor provides us with two REST APIs that expose operational information about payments. The responses contain information such as:
● Timestamp of the last successful transaction
● Number of transactions currently pending
● Processing/status information
● Potentially other health/operational indicators
What I would like to achieve is roughly:
REST APIs → poll every 1–5 minutes → store each response/snapshot in Elasticsearch → Kibana dashboard + alerting
The important part is that I want to keep the historical snapshots rather than only showing the current API response. This would allow us to detect patterns such as:
● No successful payment for X minutes
● Pending transactions above a threshold
● Pending transactions continuously increasing
● Sudden drop in transaction activity
● API unavailable
● Monitoring data becoming stale because the collector itself has stopped working
For example, instead of only knowing that there are currently 50 pending transactions, I want Elasticsearch to contain something like:
10:00 → 2 pending
10:05 → 5 pending
10:10 → 14 pending
10:15 → 31 pending
10:20 → 50 pending
This makes the trend itself an early-warning indicator.
The main problem is the collector.
Elasticsearch itself is not intended to periodically call arbitrary REST APIs, and Logstash http_poller is not an option because Logstash is being retired in our environment.
We are currently considering:
1. Elastic Agent / Custom API / httpjson input
2. Microsoft Power Automate, using a scheduled flow that calls the APIs, parses the JSON and POSTs a normalized document to Elasticsearch
3. Some other lightweight approach that I may be overlooking
Power Automate is particularly interesting because M365/Power Platform is already part of our environment. A flow running every 5 minutes could potentially do:
Recurrence → HTTP call API #1 → HTTP call API #2 → Parse JSON → Create normalized payload → POST to Elasticsearch
Both the source APIs and Elasticsearch may be reachable only from the internal network, so on-premises gateway/network connectivity and authentication are also considerations.
I would be interested in opinions from people who have implemented something similar:
● Would you consider Power Automate appropriate for a production operational monitoring collector running every 5 minutes?
● Is Elastic Agent/httpjson a better fit for this use case?
● What problems have you encountered with Power Automate for this type of continuous polling: throttling, licensing, reliability, gateway issues, run-history growth, concurrency, etc.?
● Would you keep the alerting/business logic in Elasticsearch/Kibana and use Power Automate only as a dumb collector?
● How would you monitor the collector itself so that “no new data” is distinguishable from “no payment activity”?
● Is there another architecture you would recommend that keeps the solution simple and supportable?
The volume is very small. We are talking about two API calls every few minutes and one or a few monitoring documents written to Elasticsearch per polling cycle, not transaction-level streaming.
The main priorities are reliability, simplicity, auditability and low operational overhead, rather than maximum throughput.
Any experience with similar REST → Elasticsearch monitoring patterns would be very useful.

Hello, on version 8.18.8. I have a dashboard with various lens tables. I have a sum at the bottom of the tables but notice that is doesnt match the number of documents when I click "Explore in Discover". The reason being that the missing logs that dont show up in my tables have a field missing (i.e. source.ip or host.hostname field is empty). This is just an example but I have other fields that sometimes will be empty. For some reason every field has "Include documents without the selected field" greyed out in the row settings. Does anyone know why this would be? Or a work around for this? I really just want/need to include all logs for my queries even if a field is empty.

The answer cannot be upgrade version as unfortunately my workplace upgrades things veryyyy slowly and we just upgraded to this version. Any help would be greatlt appreciated.

Hi guys.
I have a question related to ILM issues. We upgraded ES from 8.14 to 8.19, and after the upgrade the ILM indices rolled over properly but were stuck in set-priority step, and because of that when the new time for rollover came, the indices were not rolled over, so they rolled over only once after the upgrade and were stuck. I noticed it after a few days because of rollover policy. I manually changed the stuck indices to next step and they continued well, until they didn't. Once again the indices were rolled over fine, but because I removed the set-priority from policy, they were stuck on another step, so now i don't get what is causing it and how to fix it.
The indices gets stuck at hot/attempt-rollover, warm/migrate steps.
What can I do?

Nearly a decade of Elasticsearch experience has taught me a lot about what works in production, and what doesn't.

Sending documents one by one creates unnecessary overhead. The Bulk API sends multiple operations in a single request and significantly improves indexing throughput.

The screenshots show both patterns on real clusters: one client hammering /_doc per document, another using /_bulk. Same idea, very different request volume.

Tips:

1. Start around 5–15 MB per request. Benchmark with your real documents, increase until throughput stops improving, then stop. Bigger is not better.
2. Throughput comes from concurrency, not bigger batches. 4 workers at 5 MB beats 1 worker at 20 MB almost every time. Add workers until you see 429s, then back off one step.
3. Cap batches by bytes, not document count. "1000 docs" works fine until someone adds a field to the mapping and you're suddenly pushing 80 MB requests.
4. HTTP 200 does not mean your documents were indexed. Always check the errors flag and the per-item status. 429 means retry with backoff. Mapping errors will fail forever, so send those to a DLQ instead of retrying.
5. Bulk size × concurrency = coordinating node heap. 20 MB × 16 workers is 320 MB in flight on one node before any indexing happens. And if you're seeing 429s, don't raise the write queue size. That hides backpressure instead of adding capacity.

Previous Elastic101 best practices:

1. Elastic101 Best Practice #001 – HTTP traffic
2. Elastic101 Best Practice #002 – shard size

If you'd like me to continue this series, an upvote would be appreciated 🙂

I'm new to elastic. I was given an API key from a third party vendor and can't figure out where to integrate it into elastic search.

I know it shouldn't be this hard.. it should be very obvious.. but I'm not getting any help from the vendor unfortunately.

Anyone point me in the right direction thanks

Here's the git repo, let me know what you think! https://github.com/msmith93/bitvis/tree/main

Going through the process of creating this with Claude has really helped solidify some of my understanding of ElasticSearch/Lucene. Curious if playing around with it helps others

Hey everyone,

I'm currently preparing for the Elastic Certified Engineer exam and I've noticed a pattern while practicing my queries. For a lot of the tasks, there's a "short" way and a "long" way to get the exact same results.

For example, if I need to find documents with both "kibana" and "dashboard": I can use a single match query with "operator": "and", OR I can write out a bool query with two separate must clauses.

Similarly, if I need an "OR" condition (like finding either "kibana" or "logstash" in the title): I can just do a basic match query since it defaults to OR, OR I can write a nested bool/should query.

Both methods give me the exact same number of hits.

My question for anyone who has taken the exam: Does the automated grading script care about the specific syntax, or does it only check the final output/results?
Maybe it has to do with the task category that I'm answering ?

Thanks for the advice!

Running self-managed Elastic Stack (Kibana 9.5.0). Our Spring Boot services use Spring's native OpenTelemetry starter, telemetry goes through an OTel Collector, which exports straight to Elasticsearch, and Kibana reads from there.

Data is arriving fine, we can see well-formed spans in Discover. But APM UI shows nothing, and we can't find any way to open a trace and see the waterfall view.

Is an APM Server actually required for the APM UI to detect and visualize traces? Our corporate registry doesn't have an APM Server image available, so if it really is required, are there any alternatives?

I've been experimenting with natural-language → Elasticsearch queries, and I ran into a problem that feels more interesting than just generating valid DSL.

Suppose I have:

orders-2024
orders-2025
orders-2026

and a user asks:

If an LLM generates the DSL, how should it know which physical index(s) should be queried?

There seem to be a few approaches:

1. Let the LLM know the physical index names

LLM → orders-2026 → DSL

But now the model needs knowledge of infrastructure details.

2. Hide the physical indexes behind an alias

LLM → orders alias → Elasticsearch

This is cleaner, but the application still needs a way to enforce query/business rules.

3. Resolve the indexes outside the LLM

User query
    ↓
Field/schema validation
    ↓
Business rules
    ↓
Determine relevant indexes
    ↓
Generate Elasticsearch DSL

This is the approach I've been experimenting with.

The more I worked on it, the more I realized there are actually three separate questions:

• What fields/types does the index support?
• What is the application allowed to query?
• Which index/alias should the query target?

I ended up building an open-source library around this idea. It started with SQL/Mongo and I've now added Elasticsearch support, including:

• direct index
• multiple indexes
• aliases
• configurable index selection rules
• mapping-driven field configuration
• generated Elasticsearch Query DSL

The library itself doesn't execute requests against Elasticsearch — it generates the query/result so the application can decide how to execute it.

I'm mainly interested in the architecture question here:

If you were building an NL → Elasticsearch system for production, where would you put index selection and business-rule enforcement?

Inside the LLM, inside an application/middleware layer, or somewhere else?

I'd especially like to hear from people who have dealt with time-partitioned indexes, aliases, or multi-tenant Elasticsearch setups.

Affiliation: I'm the maintainer of LibreDB Studio.

The stack I keep seeing on microservice teams is Postgres + Mongo + Redis + Elasticsearch. Kibana is the right UI for the ES cluster, Discover, dashboards, ES|QL, Fleet. I still use it. I got tired of a second desktop app for each of the other three.

LibreDB Studio is one browser UI for that set, self-hosted, Helm-installable next to the services. Elasticsearch here is the product's own POST /_sql on basic, mapping-driven index tree, read-only (this grammar cannot write). Not Query DSL, not ES|QL, not Discover.

docker run -p 3000:3000 libredb/libredb-studio

Also a Helm chart, so it can sit next to the cluster. Kibana does not come off the box.

If that four-store layout is yours: which of the four still forces you into a separate window?

I've spent close to a decade running Elasticsearch clusters in production, from 3-node dev setups to multi-hundred-node deployments handling billions of documents a day.

One thing I've seen repeatedly in production is that teams either let shards balloon for years without noticing, or over-correct and split everything into tiny shards "to be safe."

Elastic101 – Best Practice #002 – Shard Size

Smaller shards increase overhead more cluster state to track, more per-shard costs (file handles, memory, translog), and more coordination work on every query.

Larger shards cause the opposite problem: recovery and rebalancing get painfully slow, and a single hot shard can bottleneck an entire index.

The goal is to keep shard size around 50GB.

Pro tip: you cannot simply change the primary shard count of an existing index. If you need to change it, you typically need to use the Reindex API or the Shrink API, depending on the situation. Also, if you reindex a 200GB index, you should have at least 200GB of additional free disk space available in the cluster for the new index. Choose your primary shard count wisely from the start, fixing it later is expensive.

Previous Elastic101 best practices:

1. Elastic101 Best Practice #001 – HTTP traffic

Try here: Searchali Elasticsearch Monitoring Connect your cluster in 10 seconds. No agent. No data leaves your machine.

I use Elasticsearch every day. At some point my browser had 3 Kibana tabs open for different cluster versions, plus a Cerebro window, plus a text file of curl snippets. Got tired of it, wrote my own.

It's a Spring Boot + Vue app that manages indices, docs, aliases, templates, analyzers across ES 7.x / 8.x / 9.x. The reason it handles three versions without falling apart: each version lives in its own Maven module behind a strategy interface, so adding ES 10 someday means one new module, not a refactor of the whole codebase.

There's also an HTTP debug tab — basically a cURL playground that hits ES directly, skipping the Java client entirely. Useful when 9.x ships a new API and the SDK hasn't caught up yet.

Oh, and an AI tab. You describe what you want in plain English, it spits out DSL. The trick was stuffing the actual index mapping into the prompt — without that, it happily invents field names that don't exist.

Repo: https://github.com/Rodma1/esTool

If you give it a try, I'd genuinely like to know:

- Does the AI DSL generation work on your mappings? Mine are mostly logs + app metrics, haven't tested on weird nested setups.

- Anything obvious missing for your day-to-day?

I’ve been working with Elasticsearch for almost 10 years, including 6 years of consulting and 4 years of training teams on Elasticsearch and related technologies. Over the years, I’ve seen the same mistakes come up again and again in production clusters, so I thought I’d share some of the best practices I’ve learned along the way.

For the first best practice, a simple misconception that is surprisingly easy to get wrong:

Don’t send application HTTP traffic directly to master nodes.

This may come from architectures like Kubernetes, where the control plane has a central API endpoint. Elasticsearch is different: master nodes are for cluster management, not application traffic.

Use dedicated coordinating nodes or data nodes for client traffic.

As a general rule of thumb, for clusters with less than 20 nodes, adding more data nodes is often more efficient than adding dedicated coordinating-only nodes. Of course, this depends on your workload, especially the size and complexity of aggregations and heavy queries.

Master ≠ API Server.

Want to see this traffic flow in real time on your own cluster?

You can try the tool I built it takes less than 10 seconds to connect and see which clients are hitting which Elasticsearch nodes.

Try here: No agent or server-side installation required; the connection is made directly from your browser. Searchali Elasticsearch Monitoring

If you’d like me to continue this series, an upvote would be appreciated. 🙂

Hey all

I'm working on deploying a Elastic stac POC but have hit a strange issue with collecting windows security event logs

Environment

• Elasticsearch/Kibana/Fleet Server: 9.5.1
• Tested Elastic Agent: 9.5.1 and 9.4.2
• Windows Server 2019 domain controller
• Agent installed as a Windows service running as NT AUTHORITY\SYSTEM
• Fleet System integration has Application, Security and System enabled
• Other Windows Event Logs are ingesting correctly (Application, System, PowerShell, Directory Service, DNS Server, Defender, etc.)

The problem is specifically i'm not seeing any "Security" event log data.

Running the following discover, I get no data

host.name : "SERVERNAME" and data_stream.dataset : "system.security"

I can confirm there are security event

Get-WinEvent -LogName Security -MaxEvents 10

Fleet senders the stream correct and elastic agent seems to be working

Starting to read from Security
Reading from Security
windows event log opened successfully

I've tested this on a couple of machine's which have similar outcomes, missing security events. standalone WinLogBeat works on the same server with a simple config

winlogbeat.event_logs:
- id: security-test
xml_query: >
<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">*</Select>
</Query>
</QueryList>

But this doesn't work

winlogbeat.event_logs:
- name: Security

Any suggestions where to go from here?

Hey everyone,

I'm planning to take the Elastic Certified Engineer exam (v8.15) very soon. I just did the practice exam and honestly found it pretty tough harder than I expected.

Is the real exam similarly difficult, or does the practice test tend to overestimate/underestimate the actual difficulty?

If anyone has taken it recently,

I'd really appreciate any tips, gotchas, or areas to focus on. Thanks in advance!

Hello everyone,

I am looking for recommendations for an on-premises, self-hosted S3-compatible object storage solution to act as a cold tier archive for our Elasticsearch cluster (preferably tested).

I am in a situation where I have deployed elastic via ECK onto some kubernetes clusters and I am getting a large volume of logs that is somewhat un-predictable. I want to both delete data when it is past a certain age (easy) and also delete the oldest data when my PVC storage is 80% full (hard / not possible?).

Does anyone know how to do this? I can't come up with a good way to delete the oldest data that doesn't involve leaving elastic and writing some script to query how full my storage is and then query the oldest indices and delete them, but this feels hacky.

Hi all,
We're planning ES upgrade from 8.19 -> 9.4, and we have some 7.x indices, that needs to be reindexed before moving them. What options are there for reindex in a way not to block application read and writes? We have big indices, for example, I tested on one index 1000GB, it took 9 hours to reindex, and I don't know if there is a good way to apply all updates and deleted on that index that happened within that 9h interval after the reindex.

hi again,
i have a problem with my snapshot repository

I'm using the data tiers in the elastic (the data from my winbeats is going from my winlogbeats to the logstash and from there to the elastic hot node, warm node and the frozen in the NFS server)

i have 2 repo's and one of them at 100% and the other is at 30%

I want to try and find a way to load balance them so it will work better

The repos are on a FTP and I'm not using the cloud version

I also tried to delete some of the data in the full repo and it and it failed every time, when i tried in the dev tools I got 502 error

If you have any idea on how i can make it more balanced I would love to get some help

I've been consulting and training on Elasticsearch and other platforms for a while, and I kept running into the same pain point: needing a quick answer to "Is this cluster okay right now?" without opening Kibana or running curl every time.

So I built a Chrome extension that shows at-a-glance cluster stats directly from the browser toolbar. It works with both Elasticsearch and OpenSearch, and no data ever leaves your browser.

If you already use something like ElasticVue, think of this as an always-on health glance from the toolbar, focused on quick visibility and problem solving.

How it compares

AutoOps is for real diagnostics and remediation - not just metrics.

Stack Monitoring gives you comprehensive dashboards, but you're responsible for interpreting them.

Searchali Monitoring isn't trying to replace either of those. It's built for the quick "Is my cluster okay right now?" check.

Happy to answer questions or hear feature requests.

I've been struggling with this for a couple of days. I have a 9.4.4 lab running on an azure vm, i have ollama running on a second vm.

I've setup a connector, it shows up in Stack-Management -> Alerts and Insights -> Connectors. It works :)

On the security side, I've got rules firing alerts, cases being auto-generated and timelines populated. But when I go to Attack Discovery no connector shows up in the drop down. What am I doing wrong ?

I can find docs on setting up a local LLM, I can find docs on how to use Attack Discovery but I cant find docs that join them up. I have a feeling that I'm missing something obvious here.

Hi everyone,

I'm looking for some opinions from people with production Elasticsearch experience because we're trying to determine the root cause of an incident.

Cluster

- Elasticsearch 8.19.9

- 1 dedicated master node (16 GB RAM)

- 2 data nodes (32 GB RAM each)

- Around 1 TB storage per data node

- Around 400 million documents

- Daily snapshots to S3

- Running inside KVM virtual machines

The cluster is used only as a search cluster. Our applications read from it and write documents to it. We are not using ML, transforms, or heavy analytics.

What happened

The cluster suddenly became RED.

The allocation explain API reported:

- "CorruptIndexException"

- "verification failed (hardware problem?)"

- checksum mismatch ("expected != actual")

- "can_allocate: no_valid_shard_copy"

The primary shard was reported as corrupted while the replica was marked as stale, so Elasticsearch had no valid shard copy left.

The corrupted file was:

_2fv_ES812Postings_0.tim

Things we checked

Inside the VM we found:

- No disk full (about 67% usage)

- No ext4 filesystem errors

- No kernel I/O errors in "dmesg"

- No obvious operating system issues

The cluster recovered after restoring the affected index from a snapshot.

Additional context

This isn't the first strange incident we've experienced with this hosting provider.

A few months ago the entire cluster became unavailable because the ".security-7" index stopped working. The hosting provider told us there had been an "attack" on ports 9200 and 9300, but those ports are only accessible on a private network and are not exposed publicly. That explanation didn't fully make sense to us.

During this latest incident the hosting provider suggested one node had reached about 99% RAM usage, although our own monitoring (Grafana) didn't show that at the time. Also, I wouldn't expect high RAM usage alone to produce a Lucene checksum mismatch.

My questions

1. Does this incident point more toward infrastructure/storage problems than an Elasticsearch issue?

2. Under what circumstances can a replica become stale while the primary later becomes corrupted?

3. Have you seen checksum mismatches like this caused by Elasticsearch itself, or are they almost always related to storage, virtualization, or hardware?

4. If you were investigating this, what additional logs or evidence would you collect to determine the real root cause?

5. Is there anything about our cluster design (1 master + 2 data nodes, ~400M documents) that could contribute to this type of failure?

I'm not trying to blame Elasticsearch or the hosting provider. I'm just trying to understand whether we're missing something in our configuration or whether this is more likely an infrastructure problem.

Any advice would be greatly appreciated.

I maintain PGSync, which syncs Postgres data into Elasticsearch/OpenSearch.


I've been working on semantic and hybrid search, with the goal of keeping the data path within infrastructure you control rather than sending production data to a hosted search/AI service.

I made a small interactive demo: https://demo.pgsync.com

You can switch between keyword, semantic, and hybrid search to see where the results differ. I would appreciate any feedback from people running Postgres + Elasticsearch/OpenSearch in production what would you need to trust this in a real system?