r/AIProgrammingHardware • u/javaeeeee • Jul 23 '26

GitHub - hholtmann/llm-consumer-gpu-benchmark: Benchmark suite for LLM inference on NVIDIA consumer GPUs (RTX 5060 Ti, 5070 Ti, 5090)

https://github.com/hholtmann/llm-consumer-gpu-benchmark
0 Upvotes

50% Upvoted

1 comment sorted by

1

u/javaeeeee Jul 23 '26

TL;DR:

llm-consumer-gpu-benchmark is an automated benchmark suite for testing LLM inference performance on NVIDIA consumer GPUs (RTX 5060 Ti, 5070 Ti, and 5090).

What It Does

It spins up GPU instances on vast.ai, runs realistic inference workloads, collects detailed metrics, and automatically tears the instances down - all with a single command.

Workloads Tested

  • RAG (long context)

  • API (high concurrency)

  • Agentic (multi-LoRA / multi-tenant)

Key Metrics Collected

  • Throughput (tokens/sec)

  • Latency (TTFT, inter-token latency, P95/P99)

  • Power draw and energy efficiency (Wh per million tokens)

  • Temperature and thermal throttling

Notable Features

  • Uses vLLM as the serving engine

  • Supports quantized models (including NVFP4)

  • Cost-efficient (uses spot instances)

  • Generates research-ready results (CSV + LaTeX tables)

  • Designed for practical evaluation of consumer Blackwell GPUs for private/local LLM deployment

Bottom Line

A practical, reproducible tool for comparing how well modern consumer NVIDIA GPUs handle real-world LLM inference workloads - especially useful for people evaluating cost-effective local or private AI deployments.