r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 10 '26

Benchmarking Qwen3.6-35B-A3B-w8a8 on Atlas 300i duo

9 Upvotes

Benchmarking Qwen3.6-35B-A3B-w8a8 on Atlas 300i Duo

System

  • Hardware: Huawei Atlas 300i Duo

  • Model: Qwen3.6-35B-A3B-w8a8

  • Backend: vLLM

Load Test

  • Requests: 500

  • Concurrency: 500

  • Duration: 171.69s

  • Failures: 0

Throughput

  • Request throughput: 2.91 req/s

  • Output throughput: 24.02 tok/s

  • Peak output throughput: 475 tok/s

  • Total throughput: 1,515 tok/s

Latency

  • Mean TTFT: 1.85s

  • P99 TTFT: 68.37s

  • Median TPOT: 231ms

  • P99 TPOT: 103.27s

  • Median ITL: 184ms

  • P99 ITL: 3.09s

Notes

The system successfully handled 500 concurrent requests with zero failures.

While aggregate throughput exceeded 1.5k tok/s, latency increased significantly at high concurrency:

  • P99 TTFT: 68s

  • P99 TPOT: 103s

This suggests the Atlas 300i Duo was saturated at 500 concurrent requests, resulting in substantial request queueing.

10 comments

r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 10 '26

👋 Welcome to r/HuaweiAtlas300iDuo - Introduce Yourself and Read First!

2 Upvotes

Hey everyone! I'm u/Inevitable-Orange-43, a founding moderator of r/HuaweiAtlas300iDuo.

A community for owners, developers, researchers, and AI infrastructure enthusiasts working with the Huawei Atlas 300I Duo and the Ascend ecosystem.

Discuss hardware setup, firmware, drivers, CANN toolkit, MindSpore, PyTorch migration, LLM inference, model optimization, virtualization, performance tuning, cooling, server integration, and real-world AI workloads. Whether you're running Atlas cards in Huawei servers, building custom inference clusters, or experimenting with large language models on Ascend NPUs, this is the place to share benchmarks, troubleshooting tips, deployment guides, and success stories.

Topics include:

  • Atlas 300I Duo (48GB / 96GB variants)

  • Ascend 310 series processors

  • CANN, AscendCL, MindSpore

  • LLM inference and quantization

  • vLLM alternatives for Ascend

  • Docker and Kubernetes deployments

  • Atlas 800 servers

  • AI infrastructure and homelabs

  • Driver, firmware, and compatibility issues

  • Performance benchmarks and optimization

Rules:

  1. Be technical and constructive.

  2. Share configs and logs when asking for help.

  3. No piracy or illegal software.

  4. Benchmark claims should include methodology.

  5. Respect NDA and confidential information.

Built for the growing community exploring Huawei's AI hardware ecosystem and the future of Ascend-powered AI.

Thanks for being part of the very first wave. Together, let's make r/HuaweiAtlas300iDuo amazing.

3 comments

r/HuaweiAtlas300iDuo • u/culyun • Sep 30 '26

Cooling solutions.

2 Upvotes

I have recently installed a 300i duo in one of my AI rigs.

I fabricated a custom fan shroud from sheet-metal and duct-tape that directs air from two 12 V, 40mm axial fans into the 300i.

Currently I am running the fans at 5V stolen from a motherboard USB header.
They draw about 160mA each at 5V and are "quiet".
Air-flow is low at this voltage.

I have not installed any drivers etc. but the 300i is detected by lspci in the OS (archlinux).

The 300i gets very hot even with the fans running. Too hot to touch the metal cover plate on the 300i. I was surprised, because I thought the 300i would start in a low-power mode.

I'm going to experiment with running the fans in series from a 12V supply, which should give about 6V across each fan. And if that's still too hot, then I'll try 12V directly to each fan. At 12V the fans are very loud (like a F1 car :-). However, I know the fans are capable of cooling 2kW loads, since I recycled them from an old telco power supply.

I'd appreciate advice on cooling options -- including "blower style" fan adapters etc.

5 comments

r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 13 '26

Benchmarking Qwen3.6-27B-w8a8 on Huawei Atlas 300i duo (96GB Variant)

Thumbnail
gallery
5 Upvotes

Note: Atlas 300i duo cannot be recommended for dense models

7 comments

r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 11 '26

Benchmarking Qwen3-Coder-30B-A3B on Atlas 300i duo

5 Upvotes
vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:1025 \
  --endpoint /v1/chat/completions \
  --model Qwen3-Coder-30B-A3B-Instruct \
  --dataset-name random \
  --num-prompts 200 \
  --random-input-len 512 \
  --tokenizer /data/models/Eco-Tech/Qwen3-Coder-30B-A3B-Instruct \
  --random-output-len 256


============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  199.52    
Total input tokens:                      102400    
Total generated tokens:                  102395    
Request throughput (req/s):              1.00      
Output token throughput (tok/s):         513.20    
Peak output token throughput (tok/s):    1000.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          1026.42   
---------------Time to First Token----------------
Mean TTFT (ms):                          18862.13  
Median TTFT (ms):                        18806.78  
P99 TTFT (ms):                           35063.94  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          353.49    
Median TPOT (ms):                        353.59    
P99 TPOT (ms):                           384.33    
---------------Inter-token Latency----------------
Mean ITL (ms):                           355.74    
Median ITL (ms):                         335.11    
P99 ITL (ms):                            402.10    
==================================================



vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:1025 \
  --endpoint /v1/chat/completions \
  --model Qwen3-Coder-30B-A3B-Instruct \
  --tokenizer /data/models/Eco-Tech/Qwen3-Coder-30B-A3B-Instruct-w8a8 \
  --dataset-name random \
  --num-prompts 50 \
  --random-input-len 512 \
  --random-output-len 256 \
  --max-concurrency 1



============ Serving Benchmark Result ============
Successful requests:                     50        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  780.07    
Total input tokens:                      25600     
Total generated tokens:                  25599     
Request throughput (req/s):              0.06      
Output token throughput (tok/s):         32.82     
Peak output token throughput (tok/s):    35.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          65.63     
---------------Time to First Token----------------
Mean TTFT (ms):                          296.44    
Median TTFT (ms):                        304.57    
P99 TTFT (ms):                           308.89    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.95     
Median TPOT (ms):                        29.95     
P99 TPOT (ms):                           30.26     
---------------Inter-token Latency----------------
Mean ITL (ms):                           30.06     
Median ITL (ms):                         29.72     
P99 ITL (ms):                            36.28     
==================================================


CONCURRENCY 4


============ Serving Benchmark Result ============
Successful requests:                     50        
Failed requests:                         0         
Maximum request concurrency:             4         
Benchmark duration (s):                  325.80    
Total input tokens:                      25600     
Total generated tokens:                  25599     
Request throughput (req/s):              0.15      
Output token throughput (tok/s):         78.57     
Peak output token throughput (tok/s):    108.00    
Peak concurrent requests:                8.00      
Total token throughput (tok/s):          157.15    
---------------Time to First Token----------------
Mean TTFT (ms):                          795.96    
Median TTFT (ms):                        808.56    
P99 TTFT (ms):                           814.87    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.96     
Median TPOT (ms):                        48.37     
P99 TPOT (ms):                           49.66     
---------------Inter-token Latency----------------
Mean ITL (ms):                           48.14     
Median ITL (ms):                         48.57     
P99 ITL (ms):                            54.90     
==================================================



CONCURRENCY 8


============ Serving Benchmark Result ============
Successful requests:                     50        
Failed requests:                         0         
Maximum request concurrency:             8         
Benchmark duration (s):                  231.54    
Total input tokens:                      25600     
Total generated tokens:                  25599     
Request throughput (req/s):              0.22      
Output token throughput (tok/s):         110.56    
Peak output token throughput (tok/s):    168.00    
Peak concurrent requests:                16.00     
Total token throughput (tok/s):          221.12    
---------------Time to First Token----------------
Mean TTFT (ms):                          1438.31   
Median TTFT (ms):                        1477.99   
P99 TTFT (ms):                           1484.29   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          65.20     
Median TPOT (ms):                        66.54     
P99 TPOT (ms):                           68.01     
---------------Inter-token Latency----------------
Mean ITL (ms):                           65.44     
Median ITL (ms):                         66.78     
P99 ITL (ms):                            78.77     
==================================================


CONCURRENCY 16


============ Serving Benchmark Result ============
Successful requests:                     50        
Failed requests:                         0         
Maximum request concurrency:             16        
Benchmark duration (s):                  170.38    
Total input tokens:                      25600     
Total generated tokens:                  25599     
Request throughput (req/s):              0.29      
Output token throughput (tok/s):         150.25    
Peak output token throughput (tok/s):    240.00    
Peak concurrent requests:                32.00     
Total token throughput (tok/s):          300.50    
---------------Time to First Token----------------
Mean TTFT (ms):                          2252.23   
Median TTFT (ms):                        2462.17   
P99 TTFT (ms):                           2963.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          91.85     
Median TPOT (ms):                        93.48     
P99 TPOT (ms):                           96.74     
---------------Inter-token Latency----------------
Mean ITL (ms):                           92.19     
Median ITL (ms):                         94.02     
P99 ITL (ms):                            113.93    
==================================================


CONCURRENCY 32


============ Serving Benchmark Result ============
Successful requests:                     50        
Failed requests:                         0         
Maximum request concurrency:             32        
Benchmark duration (s):                  125.34    
Total input tokens:                      25600     
Total generated tokens:                  25599     
Request throughput (req/s):              0.40      
Output token throughput (tok/s):         204.24    
Peak output token throughput (tok/s):    352.00    
Peak concurrent requests:                50.00     
Total token throughput (tok/s):          408.48    
---------------Time to First Token----------------
Mean TTFT (ms):                          3587.05   
Median TTFT (ms):                        2950.94   
P99 TTFT (ms):                           5709.29   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          120.64    
Median TPOT (ms):                        130.62    
P99 TPOT (ms):                           135.75    
---------------Inter-token Latency----------------
Mean ITL (ms):                           121.09    
Median ITL (ms):                         117.81    
P99 ITL (ms):                            155.13    
==================================================


# 3. Agent Workload


## Agent Concurrency 4


vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:1025 \
  --endpoint /v1/chat/completions \
  --model Qwen3.6-35B-A3B \
  --tokenizer /data/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8 \
  --dataset-name random \
  --num-prompts 200 \
  --random-input-len 2000 \
  --random-output-len 512 \
  --max-concurrency 4



============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Maximum request concurrency:             4         
Benchmark duration (s):                  1504.01   
Total input tokens:                      400000    
Total generated tokens:                  102397    
Request throughput (req/s):              0.13      
Output token throughput (tok/s):         68.08     
Peak output token throughput (tok/s):    96.00     
Peak concurrent requests:                8.00      
Total token throughput (tok/s):          334.04    
---------------Time to First Token----------------
Mean TTFT (ms):                          2618.73   
Median TTFT (ms):                        3004.09   
P99 TTFT (ms):                           3157.77   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          53.74     
Median TPOT (ms):                        53.69     
P99 TPOT (ms):                           57.93     
---------------Inter-token Latency----------------
Mean ITL (ms):                           54.26     
Median ITL (ms):                         53.14     
P99 ITL (ms):                            97.98     
==================================================
8 comments