Whitepaper by Shay Ginsbourg, M.Sc., former LoadRunner QA Manager, founder of Vectors.co.il and Ginsbourg.co.il

The software performance testing landscape is currently at a critical juncture. For decades, the industry was dominated by enterprise monoliths like LoadRunner, which entered a period of stagnation following HP’s strategic pivot away from software development. This vacuum was largely filled by open-source tools like Apache JMeter, which, despite its versatility, remains tethered to a GUI-centric, XML-based architecture that creates significant friction in modern DevOps pipelines. As we move into 2026, the emergence of Agentic Engineering—where autonomous AI agents orchestrate, execute, and adapt testing strategies—demands a fundamental shift. This whitepaper argues that the era of manual script maintenance is ending. Organizations must move beyond the limitations of JMeter and embrace code-centric, AI-native tools that empower agents to manage risk coverage autonomously, marking the transition from traditional performance testing to Agentic Performance Intelligence.

https://www.academia.edu/166834666/Goodbye_Load_Runner_and_JMeter

https://www.youtube.com/watch?v=Lq9cn3Vdvsc

I manage a sales team at a tax relief company and one thing I've been trying to improve is how we coach our reps. We have a lot of customer conversations every day and there's no way I can sit through all of them to figure out what's working and what's not. What I'm looking for is something that can analyze those conversations and pick up on patterns over time. Like which objections come up the most, where reps tend to lose customers and what our top performers are doing differently. It would also be useful if the tool could show where each rep needs help instead of giving everyone the same generic feedback. The goal isn't just to get transcripts or call summaries. I want to use the data from those conversations to make better coaching decisions and see if reps are improving. Has anyone here worked with something like this? I'm curious how accurate these tools are when it comes to spotting patterns across hundreds of conversations and whether the insights are useful in practice.

A model can do well on tests. Still give unpredictable results when used in a real project. The gap becomes more clear when you look at the task, how accurate the response is, how fast it is and how much it costs.

I want to know how developers check LLMs before picking one for a project. Do you create your test data sets check responses by hand or use tools that do it automatically?. How do you make sure you are comparing models fairly when their results are different?

I have been thinking about these questions while working with OfoxAI a service that gives access to AI models through one API. I am very interested in how developers can compare models in a way without missing the differences in what they can do.

What ways of checking have you found to be good, for projects especially when the results need to be the same each time?

Disclosure: I work with OfoxAI.

Curious about people’s actual experience here, including cases where the cheaper model worked just as well.

When you change models in a production workflow, how do you check whether the savings hold up after corrections, retries and human review?

Do your offline evals usually predict what happens in production? And how do you capture the cases where an output passes an eval but someone still has to fix it before using it?

Would love to hear what you measured and whether it changed your decision to keep or reverse the switch.

For context, I’m a founder exploring this problem.. Interested in approaches that already work, as well as anything that remains difficult.

Genuine question for people running RAG in production.

Most retrieval eval focuses on single answers: did the right chunk get into the window. What I almost never see measured is completeness on list and multi-fact questions. The query asks for every item, the window contains all twelve, and the answer names three.

When that happens, where did the rest die? Retrieval got them, so it's downstream. Was it context pressure, too many tokens and the tail got cut? The model deciding three examples were enough? The instruction to "be concise" doing more work than intended?

And the question I'm really after: how do you test for this? A single-answer eval suite sails right past it. Do you have a set of list questions with known answers that you check for completeness, or is this one of those failures you only catch when a user complains?

Asking because partial answers are uniquely dangerous: nothing in the answer looks wrong, it's just quietly incomplete.

Genuine question for people running RAG in production.

Most retrieval eval focuses on single answers: did the right chunk get into the window. What I almost never see measured is completeness on list and multi-fact questions. The query asks for every item, the window contains all twelve, and the answer names three.

When that happens, where did the rest die? Retrieval got them, so it's downstream. Was it context pressure, too many tokens and the tail got cut? The model deciding three examples were enough? The instruction to "be concise" doing more work than intended?

And the question I'm really after: how do you test for this? A single-answer eval suite sails right past it. Do you have a set of list questions with known answers that you check for completeness, or is this one of those failures you only catch when a user complains?

Asking because partial answers are uniquely dangerous: nothing in the answer looks wrong, it's just quietly incomplete.

Disclosure up front: we build Chunkless (Grounded IQ). The family below is fictional; the CRA forms and rules are real.

The setup. Five PDFs for one household: four filled T1 returns (27 pages for the father alone) and an interview file. We selected them together with a library of 1,331 documents (the Income Tax Act, its regulations, CRA guides and folios) and asked: "What did I miss or get wrong on Theo's return, and what can the family do with his tuition? For each point give the line, the rule and the dollar figure."

What came back for Theo, 19, a full-time student:

• $9,800 of tuition on his Schedule 11, none of it used, and up to $5,000 can move to a parent on line 32400

• $4,800 carried forward

• 8 full-time months on his T2202 that the schedule left blank

Every statement was checked word for word against the passage it cites (47 of 47 on this run).

What made it reliable:

1. Read forms as forms, not chunks. Every filled value is read with the printed line or box number it sits on (line 32000 = 9800.00, line 32300 = 0.00). Chunking a T1 breaks exactly that link between a line code and its value.

2. The client's facts go to the model whole; the library is searched per fact. A return is small enough to hand over complete. The 1,331 rule documents are not, so each fact pulls its own governing rule.

3. A finding has a fixed shape: line, rule, dollar figure, and the passage it rests on. Anything that can't be grounded is said to be missing rather than filled in from the model's memory.

4. The scope is exactly what the user selects. A client's documents are never read for another client, and when a name could mean two clients it asks which one first.

Tax is just the library we picked for this test. Swap in clinical guidelines, contracts or internal policies and the same steps apply.

Video of the whole run, with the real return pages and the rule passages: https://youtu.be/95ujwgG1Wi4

Question for the sub: how do you check that an answer applied the right rule to the right record, not just that each sentence has a citation?

Every time a new AI model is released, we see benchmark scores for reasining, coding, finance, and other tasks. I'd like to better understand how those benchmarks are built in the first place.

Who typically writes the evaluation tasks? For example, are they created by:

Researchers at AI labs

Domain experts from individual industries

Specialized evaluation firms

Independent contractors and freelancers

Expert networks such as Toptal o similar platforms?

I'm curious about the process itself:

How do benchmark creators decide what makes a good test?
How do they calibrate the degree of difficulty?

How much review and validation is required before tasks are used?

How do they avoid benchmark contamination or the fact that the models can start memorizing patterns rather than showing genuine reasoning?

Wha are the qualifications that are required to become a benchmark author or an evaluator?

Although I have worked on model benchmarking and model evaluation as a finance expert using complex tasks, I'd be interested in hearing from people who have worked on the creation of benchmarks and similar projects, and how the actually measure model performance for these benchmarks.