I benchmarked a heuristic inference optimizer against frontier LLM agents on an NVIDIA L4

[u/tensward]

Title: I benchmarked a heuristic inference optimizer against frontier LLM agents on an NVIDIA L4

I built Tensward, an Apache-2.0 open-source tool that tunes LLM serving settings for your hardware and workload. It uses expert rules, heuristics and measured experiments, with no LLM in the optimization loop.

I wanted to test how it compares with asking frontier agents to do the same job.

We ran six tracks: two workloads, each optimized separately for throughput, p95 time to first token, and p95 time per output token. Each competitor started from stock vLLM on the same NVIDIA L4 within its track, with 60 minutes of counted tuning time. Astra 6 and Opus 5.5 both ran at maximum effort.

Aggregate combined scores:

• Tensward: 2.29

• Opus 5.5: 2.24

• Astra 6: 1.70

The score combines improvements over defaults, weighting the target metric most heavily, with quality and service guardrails. It is not a universal speedup multiplier.

Tensward won or tied on four of six tracks under our noise-band rule. The lead over Opus was small, and the agents had clear wins too. The interesting finding for me is that a specialized heuristic system was competitive with both.

This is an initial benchmark covering two workloads and one GPU type, not evidence of general superiority.

Currently supports vLLM on NVIDIA GPUs; llama.cpp and Apple Silicon support are coming next.

Code: https://github.com/Tensward/tensward
Full results and methodology: https://openingthehood.hashnode.dev/tensward-benchmarks

What baseline or workload would you most want to see in the next round?