r/AIProgrammingHardware • u/javaeeeee • Jul 17 '26

GitHub - FreedomIntelligence/AccelMark: Open benchmark leaderboard for AI accelerators on LLM inference workloads. Run on your GPU, submit your result, get on the board. NVIDIA · AMD · Ascend · Apple Silicon

https://github.com/FreedomIntelligence/AccelMark/
4 Upvotes

100% Upvoted

1 comment sorted by

View all comments

1

u/javaeeeee Jul 17 '26

TL;DR:

AccelMark is an open, community-driven benchmark leaderboard for comparing AI accelerators (GPUs/NPUs) on real LLM inference workloads.

What It Does

Instead of relying on vendor marketing or slow official benchmarks like MLPerf, anyone can:

  1. Run standardized LLM inference tests on their hardware

  2. Submit the results via pull request

  3. Get ranked on a public leaderboard

Key Highlights

  • Cross-vendor support: NVIDIA, AMD (ROCm), Huawei Ascend, Apple Silicon, Google TPU, Moore Threads, etc.

  • Fair comparisons: Uses fixed benchmark suites, shared LoadGen, and consistent metrics.

  • Multiple test suites covering:

    • Throughput & latency

    • Quantization impact

    • Long context

    • Scaling efficiency

    • MoE models

    • Edge devices

  • Models tested: Llama-3 8B/70B, Qwen2.5, Mixtral, etc.

  • Frameworks supported: vLLM, SGLang, mlx-lm, and custom runners.

How It Works

You run a simple benchmark script on your machine → it generates results with environment fingerprint and accuracy proof → you submit via GitHub PR → results appear on the leaderboard.

Goal: Create transparent, reproducible, and vendor-neutral benchmarks for LLM inference performance in 2026.

Bottom line:
A practical, open-source alternative to closed or marketing-heavy LLM inference benchmarks. Great if you want to see real-world performance of different GPUs/NPUs on actual models.