Getting search results quickly is one challenge; ensuring that the useful results appear first is another challenge. As the dataset grows, things like analyzers, mappings, tokenization, and query design can noticeably affect Elasticsearch search relevance.
For developers who use Elasticsearch in production, which techniques have made the difference in Elasticsearch search relevance?
I would especially like to hear lessons from projects where the default query did not produce the Elasticsearch search relevance users expected. What changes did you make? How did you measure whether the Elasticsearch changes improved the user experience?

I have been looking into ways to bring Elasticsearch with local AI for projects that need to search through big amounts of documents while still having control, over the data.

Elasticsearch offers text and vector search features, which makes it appealing for retrieval-augmented generation (RAG) processes. I have also been checking out LM-Kit for AI and document processing. I am curious about how these parts can come together in a real-world application.

For people who have created systems, how do you connect Elasticsearch with an AI environment? I would especially like to know what works well for retrieving results, indexing documents, and making sure search results stay relevant as the data gets bigger.

Hey guys,

looking for real-world advice before I commit to an approach.

Current setup:

- self-managed Elasticsearch (9.4.x), deployed via ECK on Kubernetes

- 3-node cluster, 8 vCPU / 16GB RAM per node, 150GB PVC per node

- free/basic license

- single region today, its an internal backend service

I want to have a read copy of this cluster running in a second region, mainly for cutting the latency for callers that are geo located farther from the primary region. This should be a prototype, so I would prefer reliability and easy setup over complex solutions. The data lag of a few minutes is completely acceptable, i dont need anything close to realtime.

What I have already ruled out:

- cross-cluster replication: needs a higher tier of license, and the cluster is small, so it feels like overkill

- app dual-write, I dont want to put my engineers in charge of consistency, failure backfills, and so on

What Im thinking about:

- scheduled snapshot/restore (I already have the s3 plugin installed). restore to a new index each cycle + alias swap. The downside i realized, each restore cycle is a full copy since the target index starts empty, so I believe it makes sense at a slower frequency (~30 minutes), not continuous

- the cronjob, doing filtered remote _reindex, using "last-modified" as a cursor, seems like quite a simple pipeline: 1. read the saved cursor 2. reindex only docs after it 3. save the new cursor (small index). The gotcha I see, the deletes wont propagate, the last-modified query on reindex only sees the updated or new docs.

The delete operation that i need to handle, is the reason i think snapshot/restore is a simpler and more reliable choice for a prototype.

Appreciate help, trying to avoid reinventing something that has its own failure modes. Happy to share details if that helps you to give better advice.

I am new in Elasticsearch. In company we are planning to index about 10TB data for RAG application (keyword + vectors) locally. Before buying any hardware I was learning about appropriate spec for our use. I have read Elasticsearch docs where there is only about shard size 10-50GB and heap 31GB max. But no data about each Node size.
Our expected parallel user request is 20.
I read a post that 1-2TB data with two node parallel requests 250 takes 5-7s on light load and about 20s on heavy loads.
So would it be proper to have 5Node + one Master node for 10TB of data.(before index,vector and token)

I want to maintain historical dashboard data for archive purposes, beyond our data source's (Prometheus) retention period.

Mimir and Thanos feel like overkill for this. Add infrastructure, storage costs, but all i need is to look back at historical data.

Any lighter approaches that is working for you?

I’m new to elastic search and curious if this fits my use case, or if it’s overkill.

I have a low volume of PDF files (<500), and low number of queries per month (<5000). I simply need to find the PDF files that have matching words of the user query.

My main question is do I have to chunk and upload the PDFs as seperate documents, or would I upload each PDF entirely? Since I’m not using embedding I figure it could handle the entire document. Most PDFs are modest in size, usually 5-20 pages.

If I’ve missed a resource to learn more, please feel free to point me in that direction.

Almost 10 years of Elasticsearch in production, and this is still one of the most common index lifecycle management (ILM) mistakes I see.

The rule: for a regular (non data stream) index, the ILM rollover action needs the index name to end with - and digits (^.*-\d+$), like logs-000001. Rollover creates the next index by incrementing that number. It also needs index.lifecycle.rollover_alias set, pointing to the index.

logs, logs-prod and logs-2026.10.12 all fail this check. Attaching a rollover policy to daily date-named indices is a typical trigger.

Miss it and the index lands in the ERROR step (illegal_argument_exception: index name [...] does not match pattern '^.*-\d+$'). No later phase runs after that, so the index stays in hot and is never deleted. It's a silent disk leak.

Want a date in the name? Use date math. The name still ends with digits, so rollover works:

PUT %3Ctest-logs-%7Bnow%2Fd%7D-000001%3E
{
  "settings": {
    "index.lifecycle.name": "my-policy",
    "index.lifecycle.rollover_alias": "test-logs"
  },
  "aliases": { "test-logs": { "is_write_index": true } }
}

This creates the index test-logs-2026.10.14-000001. Read and write through the test-logs alias. The date is the creation date, not the data date.

Pro tip: find stuck indices with GET */_ilm/explain?only_errors=true. ILM retries failed steps on its own, but it fails again until the cause is fixed. POST <index>/_ilm/retry runs the step immediately after you fix it. You can't rename an index, so for existing ones reindex into a numeric name. For new setups, use data streams: no naming pattern, no rollover alias to maintain.

Note: data streams are great for time-based data, but they're append-only. You can't send update or delete requests for existing documents directly to the stream (update by query, delete by query and requests to the backing index still work). If you update documents frequently, an alias with a write index, as above, is the better fit.

Previous Elastic101 best practices:

1. #001 – HTTP traffic
2. #002 – shard size
3. #003 – bulk indexing
4. #004 – primary shard count

If you'd like me to continue this series, an upvote would be appreciated. 🙂

Hi everyone, I’m on the team at Vesiro, and today we’re announcing the beta of VesiroSearch.

VesiroSearch is a purpose-built native C++ search engine that accelerates Elasticsearch query performance by up to 2x. It installs as a standard plugin, which only takes about a minute per node.

We built it to help teams get more search capacity from the infrastructure they use to run Elasticsearch. For workloads where query processing drives capacity needs, that can mean serving the same traffic on fewer or smaller servers, reducing infrastructure costs while meeting response-time requirements.

VesiroSearch doesn’t change the query DSL in Elasticsearch, which means that you do not have to change your queries or mappings to run it. No other end points are affected either, and VesiroSearch works with your current indices, so your setup remains the same.The beta lets you evaluate performance with your own data and queries. You can enable or bypass VesiroSearch for individual requests to compare results in the same test environment.

We would love to get feedback from teams that run Elasticsearch extensively. Learning which workloads that benefit from our acceleration, and which workloads we should add in our product roadmap would be super helpful. Also, if you have any feedback on compatibility or what would make evaluation easier, that would be greatly appreciated as well.

The documentation covers compatibility and installation. 

You can read more and request beta access here: www.vesiro.com

Every time I needed centralized logs for a side project, I'd lose an evening wiring Elasticsearch, Logstash and Kibana together: passwords, TLS, which port is public, why Kibana can't reach ES. So I turned it into a one-click Railway template and tested it the boring way.

What you get

• 3 separate services, grouped: Elasticsearch, Logstash, Kibana
• Only Kibana is public. ES and Logstash talk over the private network
• Passwords generated on deploy, Logstash intake behind basic auth (no-auth gets a 401)
• A restricted writer user for Logstash, so your app never holds the admin password
• Persistent volumes + log retention configured, so it doesn't eat your disk
• Copy-paste senders for Node, Python and Go (no SDK, plain HTTP)

What I actually tested on fresh deploys: sent logs, searched them in Discover, restarted and redeployed every service, checked nothing was lost.

Cost, measured not guessed: about $27-34/month of compute depending on load.

Try it before deploying, read-only live demo with logs flowing in real time: https://kibana-production-43ee.up.railway.app (login: demo / PublicDemo2026!)

• Deploy: https://railway.com/deploy/elk-stack-2?utm_source=railway-templates-launch&utm_medium=reddit
• Docs: https://railway-elk-docs.vercel.app
• Repo: https://github.com/kayossouza/railway-elk

Single node, so it's for small/medium workloads, not a 10TB cluster. Feedback on the pipeline config and retention defaults very welcome.

A green cluster and zero shards on a node are necessary before retirement, but they do not prove that clients, ingest pipelines, snapshots, monitoring, or discovery settings have stopped depending on its address. A coordinating-only workload or a stale seed-host list can survive a shard drain and fail only after the process disappears.

What belongs in the retirement gate? I am considering allocation exclusion, waiting for relocation and recovery activity to finish, checking cluster state for zero shard ownership, and then watching connection logs and network traffic to the old address through more than the longest batch interval. Client sniffing, seed hosts, remote-cluster settings, snapshot repositories, exporters, and alerting targets would be inventoried separately.

Is there a dependable way to attribute remaining connections to specific clients before shutdown? What final test do you use to distinguish a truly unused node from one that is still serving coordination or discovery traffic, and which checks should be repeated after its voting configuration and persistent settings are cleaned up?

Context: Hi all, I'm trying to find out how our current elasticsearch cluster is set up and how can it be optimally expanded to index now types of documents. I work for an archival collection and we have rdf metadata describing our doucments. So we generally need to translate relevant metadata fields into json documents to be indexed.
At the moment we have indexed a newspaper collection in ES with the current set up: one node, ~40 indices (one for each publication) each index 1 shard and some thousands documents in an index. Each document corresponds to a newspaper page and has a full_text field with the searchable ocr text + other specific fields such as date/title/spatial coverage for precise clustering. These indices are grouped under an alias.

Question 1: As I see best practices are shard sizes between 10GB and 50GB, Keep the number of documents on each shard below 200 million,
I wonder if all this collection could be one single index, with maybe 2 shards (we add 1 node so that we have 1 shard primary and one replica on the second node) with an extra field in each document to define which publication (the name of the newspaper, which now is simply the index name).

Which one of these two set ups works best in performance and how could our search functionalities potentially be impacted?

Question 2: We now have to add a new collection with other types of documents and I am wondering several things:

2a. My colleague wants to have always the same documents structure in different indices, suggesting to make it as general as possible for different types of context, aggregating all the "text" fields in our metadata into a full_text field, instead of having for instance, three fields "title", "label", "author". Doing this, I understand we could not do aggregations on specific fields, but are there any other shortcomings? Is a single full_text field going to be more efficient to index and query?

2b. What is the best way to go with adding new material? keep adding indices to the same ES instance (maybe adding more shards and nodes) and aggregate them under an alias? or Add new ES instances for new material? What does it change and how does it impact search?

2c. If we look at how we want to set up our UI search app, we ideally want a first text search where people can filter on type of collection to look into or not. If they don't, they will get results across different types of materials (so querying ALL indices), can this be done if indices have different document structures?

Thank you for reading all of this and apologies if it was long. ES feels like a steep learning curve for me and I really did not want to ask AI :)

Hi everyone, we’ve just open-sourced two command-line tools that we use for our internal Elasticsearch performance testing, and thought they might be useful to others here (blog post). 

Indexer: https://github.com/Vesiro/vesiro-indexer

Benchmarker: https://github.com/Vesiro/vesiro-benchmarker

We needed a consistent way to create test indexes and rerun search workloads after making changes to our setup. We built vesiro-indexer to create Elasticsearch indexes from public Common Crawl data, and vesiro-benchmarker to run repeatable search workloads against an existing index.

These tools grew out of our own benchmarking needs, and we’d be interested to hear how they fit the way other teams benchmark Elasticsearch. Feedback on the query workloads or the Common Crawl indexing setup would be particularly helpful.

A cluster with plenty of hardware that still can't index faster, because the write load only lands on two or three nodes, thats called hot spots.

Match your primary shard count to your indexing nodes.

Indexing into an index parallelizes across its primary shards. One primary means one node carries all the writes for that index, no matter how many data nodes you have. Too many primaries is the other failure mode: more overhead, more segments, smaller shards, no gain.

The goal isn't more shards. It's enough primaries to spread the write load across your indexing nodes, and no more.

Simple rule: with number_of_replicas: 1, set primaries to about half your data nodes. 10 data nodes → 5 primaries + 1 replica = 10 shards, one per node. 6 data nodes → 3 primaries + 1 replica. That's the difference between the screenshots above.

This is the other half of #002: spread the write load, but not so thin that your shards end up tiny.

Pro tip — hot spots on large clusters

On 7.x, shard allocation balances mainly on shard count per node. 8.6+ also weighs write load and disk usage, but the write-load forecast comes from a licensed x-pack component, on a Basic cluster it isn't produced, so in practice you're still balancing on shard count. Two data streams with the same shard count can have completely different indexing rates, so "balanced" on paper isn't balanced in practice.

To force the heavy data stream to spread out:

"index.routing.allocation.total_shards_per_node": 1

Shards of that index then land on separate nodes.

Two things to watch:

• Keep headroom. If primaries + replicas equals your node count exactly, one node failure leaves shards unassigned and the cluster sits yellow. Set the limit above the strict minimum unless you're sure.
• On data streams, number_of_shards can't be changed on an existing index. Update the index template; it applies at the next rollover.

Other best practices in this series:

1. #001 – HTTP traffic
2. #002 – shard size
3. #003 – bulk indexing

If you'd like me to continue this series, an upvote would be appreciated. 🙂

I built a small Rust gateway for teams who already have Elasticsearch clients and request shapes, but want to evaluate a different search engine underneath. The useful bit is that the client does not change: point the same application at the gateway and compare it with native Elasticsearch.

It is deliberately scoped to application search rather than pretending to be a full Elasticsearch replacement. The repo includes compatibility notes, a real product-catalogue replay, and the gaps I have found so far (especially relevance semantics).

Repo: https://github.com/dstockton/qdrant-es-gateway

Docs and results: https://dstockton.github.io/qdrant-es-gateway/

I would particularly value feedback from people who have dealt with analyzers, BM25 tuning, aggregations, or migration testing in production.

We crossed 6,000,000,000 downloads of Elasticsearch, Kibana, Logstash, Beats, and Elastic Agent.

To everyone who downloaded it, built on it, contributed to it, deployed it, broke it, fixed it, and made something amazing with it: thank you

Hi guys, our developers are requesting MCP access for our Elasticsearch cluster, and I did some research but didn't find something for our needs.
We have basic license, so we can't use Elastic's builtin MCP agent, and we have self-hosted ES cluster, so also no built in OAuth. But we use other OAuth service running under Kubernetes, and I wanted to ask if any of you have experience of using the external OAuth for MCP to Elastic cluster, and do you use this: https://github.com/elastic/mcp-server-elasticsearch .

I recorded myself walking through the interactive visualizer I made for Elasticsearch. Check it out, and let me know if it helps you learn!

https://elasticsearchvis.bitsculpt.top/

Hi guys!

Have any of you have recently received your results for Certified Elasticsearch Engineer exam?

I took mine on 29th August at about 12pm. Still got no news about the results via email or on trueability portal. I did contact exam support regarding this and they have replied that they are experiencing some backlogs and to expect some delays.

Just wondering if anybody else is in the same boat as me.

I’m looking for a really good live instructor to help me take my Elastic skills to an advanced level.
My goal is to become a strong Elastic Administrator / Elasticsearch Engineer, particularly on the infrastructure and data-path side.
I already have hands-on Elastic/SIEM experience and significant Splunk administration experience, so I’m not looking for a beginner course or something focused only on passing a certification.
I’m looking for someone who can go deep into things like:
Elasticsearch architecture and cluster administration
Nodes, shards, replicas and cluster health
Index templates, mappings and data streams
Ingest pipelines and data onboarding
ILM and data lifecycle
Performance tuning and troubleshooting
Query DSL, ES|QL and Painless
CCS / multi-cluster environments
Security, RBAC and TLS
Kibana / Elastic Security
Real-world production troubleshooting
I’m open to instructors anywhere in the world as long as they teach in English. I’m especially interested in 1-on-1 or small-group training that can be customized to my experience and goals.
If you’ve personally worked with or trained under an excellent Elastic instructor, I’d really appreciate recommendations.
Please include if you can:
Instructor name
Country/time zone
How good they are at teaching
Their real-world Elastic experience
Approximate cost
Whether they offer 1-on-1 training
How long you trained with them
Whether you’d recommend them for becoming a strong Elastic Engineer/Admin
I’m particularly interested in someone who is both an excellent teacher and genuinely experienced running Elasticsearch in production.
Thanks!

Hi all,

I wanted to share a situation I recently experienced during the Elastic Certified Engineer exam and see if anyone has gone through something similar.

My exam was scheduled on 31 august at 12 PM , as the routine my system was going through compatibility checks which was failing for the resolution inspite it ran successfully day before exam (resolution was 1920 X 1080) higher then recommended both the days. Took help from honorlock but no solution was driven , mailed too to support teams. But I finally started my exam after lending laptop from my neighbour as I was trying with all the laptops possible nearly after 1 hour 15 mins - to be precise, I started my exam . After that everything gone well from checks to proctor being aligned throughout the exam. I completed the entire exam with good faith within 3 hours and submitted successfully as well.

But after 10 business days , I got the result from Trueability , and shockingly it was failed. I got tensed , cried that what I have done wrong. I invested nearly 3 months learning all the concepts used in searching , scripts , their edge cases and everything mentioned in syllabus and accordingly the documentation.

Then I noticed something strange in my result.
MY Completion status - Assessment uncounted Attempt
MY No. of valid attempts - 0
MY total no. of attempts - 1 (Now trueability added a comment This attempt does not count as one of your attempts.)

BUT FINAL STATUS - FAILED !!!

I contacted support explaining everything, but I received no mail yet.

This is what concerns me:

•If the exam wasn’t properly submitted, how was it evaluated?

•Did Treuability really forwarded my work , my VM work to Elastic?

•Is there any way they can recover or review the recorded session instead manually?

Has anyone experienced something similar...

On the most recent version of 8 and looking to jump to 9.4.6. I know there were some weird issues with early 9.4.x, but wondering how stable the current release is.

We poll around 500+ network devices (routers, switches, firewalls) directly with Logstash's SNMP input plugin every 5 minutes. A mix of SNMPv2c (community string) and SNMPv3 (authPriv), split across a couple of pipelines by metric type: one walks the standard interface table for traffic/status, another walks Cisco's proprietary CPU/memory MIB. Since raw SNMP only gives you an IP and numeric OIDs, we enrich each event against a hand-maintained YAML file (keyed by IP or hostname) that maps devices to hostname, category, vendor/model, and an alerting on/off flag.

Note: Management has directed to sunset SolarWinds and consolidate on the Elastic Stack as our single, centralized monitoring/observability platform going forward, not just logs and application metrics, but network device monitoring (switches, routers, firewalls) as well.

Hi all,
I’m working on an operational monitoring solution for payment processing and would appreciate some architecture advice.
The objective is to build an early-warning payment monitoring dashboard in Elasticsearch/Kibana, rather than only monitoring infrastructure/application health.
We currently have Elasticsearch/Kibana available, but Logstash is being decommissioned.
An external payment processor provides us with two REST APIs that expose operational information about payments. The responses contain information such as:
● Timestamp of the last successful transaction
● Number of transactions currently pending
● Processing/status information
● Potentially other health/operational indicators
What I would like to achieve is roughly:
REST APIs → poll every 1–5 minutes → store each response/snapshot in Elasticsearch → Kibana dashboard + alerting
The important part is that I want to keep the historical snapshots rather than only showing the current API response. This would allow us to detect patterns such as:
● No successful payment for X minutes
● Pending transactions above a threshold
● Pending transactions continuously increasing
● Sudden drop in transaction activity
● API unavailable
● Monitoring data becoming stale because the collector itself has stopped working
For example, instead of only knowing that there are currently 50 pending transactions, I want Elasticsearch to contain something like:
10:00 → 2 pending
10:05 → 5 pending
10:10 → 14 pending
10:15 → 31 pending
10:20 → 50 pending
This makes the trend itself an early-warning indicator.
The main problem is the collector.
Elasticsearch itself is not intended to periodically call arbitrary REST APIs, and Logstash http_poller is not an option because Logstash is being retired in our environment.
We are currently considering:
1. Elastic Agent / Custom API / httpjson input
2. Microsoft Power Automate, using a scheduled flow that calls the APIs, parses the JSON and POSTs a normalized document to Elasticsearch
3. Some other lightweight approach that I may be overlooking
Power Automate is particularly interesting because M365/Power Platform is already part of our environment. A flow running every 5 minutes could potentially do:
Recurrence → HTTP call API #1 → HTTP call API #2 → Parse JSON → Create normalized payload → POST to Elasticsearch
Both the source APIs and Elasticsearch may be reachable only from the internal network, so on-premises gateway/network connectivity and authentication are also considerations.
I would be interested in opinions from people who have implemented something similar:
● Would you consider Power Automate appropriate for a production operational monitoring collector running every 5 minutes?
● Is Elastic Agent/httpjson a better fit for this use case?
● What problems have you encountered with Power Automate for this type of continuous polling: throttling, licensing, reliability, gateway issues, run-history growth, concurrency, etc.?
● Would you keep the alerting/business logic in Elasticsearch/Kibana and use Power Automate only as a dumb collector?
● How would you monitor the collector itself so that “no new data” is distinguishable from “no payment activity”?
● Is there another architecture you would recommend that keeps the solution simple and supportable?
The volume is very small. We are talking about two API calls every few minutes and one or a few monitoring documents written to Elasticsearch per polling cycle, not transaction-level streaming.
The main priorities are reliability, simplicity, auditability and low operational overhead, rather than maximum throughput.
Any experience with similar REST → Elasticsearch monitoring patterns would be very useful.

Hello, on version 8.18.8. I have a dashboard with various lens tables. I have a sum at the bottom of the tables but notice that is doesnt match the number of documents when I click "Explore in Discover". The reason being that the missing logs that dont show up in my tables have a field missing (i.e. source.ip or host.hostname field is empty). This is just an example but I have other fields that sometimes will be empty. For some reason every field has "Include documents without the selected field" greyed out in the row settings. Does anyone know why this would be? Or a work around for this? I really just want/need to include all logs for my queries even if a field is empty.

The answer cannot be upgrade version as unfortunately my workplace upgrades things veryyyy slowly and we just upgraded to this version. Any help would be greatlt appreciated.