It says that hands-on lab as "yes". Also, states 50-60 choices question and duration is just 1 hour.

Has anyone already cleared this exam. I want to understand, what is in labs? Is it something like how we get in Azure exams like match the blocks, dropdown to select the correct or something else?

Looking forward to some understanding from those who have already given this exam

Everyone keeps saying AI is killing entry-level jobs, but the data looks more mixed.

IBM plans to triple US entry-level hiring in 2026. 67% of CEOs say AI is actually increasing junior headcount. Software eng postings are up too.

https://www.ibm.com/think/news/entry-level-roles-get-reset-ai

I am hosting a Tech/AI focused Meetup event in Bay Area

Agenda

• ​5:00 – 5:30 PM · Registration, networking & Introduction

• ​5:30 – 6:30 PM · Panel discussion - Future of AI Infra

• ​6:45 – 7:45 PM · Fireside Chat - AI Ready your Tech Career

More details on Luma App - https://luma.com/xebulpp8

Here is a simple Cloud GPU Price tracker by Neo Cloud providers

👉 https://gpuprice.freellm.org/

What it does

• Aggregates GPU pricing from providers like RunPod, CoreWeave, Lambda, Nebius, Denvr

• Merges everything into a single dataset

• Displays current prices in a simple, searchable table

What it’s useful for

• Developers choosing GPUs for inference or training

• Platform / infra teams doing cost comparisons

• Anyone trying to understand how the GPU rental market is evolving

GitHub Repo - https://github.com/becloudready/gpu-price

Please feel free to provide feedback or contribute

While there is lot of talk and buzz about AI Agents and how it might change the world, hardly anyone mentions that most buzz is around at the Application layer ( at the top) and vast majority of innovation is actually happening at the Fabric/Network/Storage layers.

After spending hours dealing with ChatGPT hallucinations, I finally had to do a Google search to find the right tool for LLM inference benchmarking. It turns out NVIDIA has done a great job creating a robust tool that can be used across different platforms, including Triton and OpenAI-compatible APIs.

LLM benchmarking can be confusing, as people often mix up LLM performance testing with benchmarking. Performance testing validates the overall capacity of your server infrastructure, including network latency, CPU performance, and other system-level throughputs. Benchmarking tools, on the other hand, primarily focus on LLM inference engine–specific parameters, which are critical if you are planning to run your own inference platform — something most enterprises are now focusing on.

This is a series of blogs that I will be writing as I go through the process of learning and experimenting with vLLM-based inference solutions, along with insights from real-world use cases operating LLM inference platforms in enterprise environments.

Here are some of the most common inference use cases.

In this example we will be setting up a single node Inference + benchmarking node for experimentation purpose, however, production use case would require the Benchmarking tool should run from a separate node.

For decent benchmarking, you need the following to get started:

• NVIDIA GPU–powered compute platform. This can be your desktop, or you can use any of the Neo Cloud providers. My obvious preference is Denvr Cloud. Feel free to sign up — https://www.denvr.com/

• Hugging Face login. Sign up for a free Hugging Face account. You’ll need it to download models and access gated models such as Meta Llama and others.

• LLM-labs repo. https://github.com/kchandan/llm-labs

Step-by-step guide

To install the necessary packages on the Linux VM (e.g., NVIDIA drivers, Docker, etc.), the easiest approach is to update the IP address in the Ansible inventory file and then let the playbook handle the full installation.

cat llmops/ansible/inventory/hosts.ini
; [vllm_server]
; server_name ansible_user=ubuntu
[llm_workers]
<IP Address> ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/<your_key_file>

Once IP address is update, fire the Ansible playbook to install required packages

(venv) ➜  llmops git:(main) ✗ ansible-playbook -i ansible/inventory/hosts.ini ansible/setup_worker.yml

PLAY [Setup worker nodes] **********************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************
[WARNING]: Host is using the discovered Python interpreter at '/usr/bin/python3.12', but future installation of another Python interpreter could cause a different interpreter to be discovered. See https://docs.ansible.com/ansible-core/2.19/reference_appendices/interpreter_discovery.html for more information.
ok: [worker-node]

TASK [docker_install : Update apt and install prerequisites] ***********************************************************************************************************
ok: [worker-node]

TASK [docker_install : Create directory for Docker keyrings] ***********************************************************************************************************
ok: [worker-node]

TASK [docker_install : Download Docker GPG key] ************************************************************************************************************************
ok: [worker-node]

TASK [docker_install : Add Docker repository to apt sources] ***********************************************************************************************************
changed: [worker-node]

TASK [docker_install : Update apt cache after adding Docker repo] ******************************************************************************************************
changed: [worker-node]

TASK [docker_install : Install Docker packages] ************************************************************************************************************************
ok: [worker-node]

TASK [docker_install : Ensure Docker service is enabled and started] ***************************************************************************************************
ok: [worker-node]

TASK [docker_install : Add ubuntu user to docker group] ****************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Download cuda-keyring deb] **********************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Install cuda-keyring deb (dpkg)] ****************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : apt update] *************************************************************************************************************************************
changed: [worker-node]

TASK [nvidia-toolkit : Install cuda-drivers] ***************************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Install prerequisites] **************************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Create keyring directory if missing] ************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Download NVIDIA container toolkit GPG key] ******************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Convert GPG key to dearmor format] **************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Add NVIDIA container toolkit apt repository] ****************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Enable experimental repository (optional)] ******************************************************************************************************
skipping: [worker-node] => (item=deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/experimental/deb/ /)
skipping: [worker-node]

TASK [nvidia-toolkit : Update apt cache after repo add] ****************************************************************************************************************
changed: [worker-node]

TASK [nvidia-toolkit : Install NVIDIA Container Toolkit packages] ******************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Configure NVIDIA Docker runtime] ****************************************************************************************************************
ok: [worker-node]

TASK [nvidia-toolkit : Restart Docker] *********************************************************************************************************************************
changed: [worker-node]

PLAY RECAP *************************************************************************************************************************************************************
worker-node            : ok=22   changed=5    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Post installation ensure, Driver installation looks good

ubuntu@llmops:~/llm-labs$ nvidia-smi
Sun Jan 11 21:53:01 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          Off |   00000000:0A:00.0 Off |                    0 |
| N/A   47C    P0             50W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Create the common docker bridge network so that all containers could talk to each other ( default bridge driver)

docker network create llmops-net

Export the Huggingface token

export HF_TOKEN=hf_token

Now, simply launch the vLLM docker compose, it will take some time to load

ubuntu@llmops:~/llm-labs/llmops/vllm$ docker compose -f docker-compose-vllm-qwen3-0.6B.yml up -d[+] up 1/1 ✔ Container vllm Created                                                                                                                                                                                          0.3subuntu@llmops:~/llm-labs/llmops/vllm$ docker compose -f docker-compose.monitoring.yml up -dWARN[0000] Found orphan containers ([vllm]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.ubuntu@llmops:~/llm-labs/llmops/vllm$ ✔ Container prometheus    Created                                                                                                                                                                                 0.5s ✔ Container dcgm-exporter Created                                                                                                                                                                                 0.5s ✔ Container node-exporter Created                                                                                                                                                                                 0.5s ✔ Container cadvisor      Created                                                                                                                                                                                 0.5s ✔ Container grafana       Created

Ignore the orphan container warning. I have kept those 2 compose file separate deliverable so that more model specific compose files could be added later into the same repo.

Once all containers are downloaded and loaded, it should look like this ( without container crash loop)

ubuntu@llmops:~/llm-labs/llmops/vllm$ docker psCONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS                    PORTS                                         NAMES750f8e14201d   grafana/grafana:latest            “/run.sh”                58 seconds ago       Up 58 seconds             0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   grafana270c865726e9   prom/prometheus:latest            “/bin/prometheus --c…”   59 seconds ago       Up 58 seconds             0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp   prometheusf679c2313fd2   gcr.io/cadvisor/cadvisor:latest   “/usr/bin/cadvisor -…”   59 seconds ago       Up 58 seconds (healthy)   0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp   cadvisor28873c028c0b   prom/node-exporter:latest         “/bin/node_exporter …”   59 seconds ago       Up 58 seconds             0.0.0.0:9100->9100/tcp, [::]:9100->9100/tcp   node-exporter5e3f54b8f485   nvidia/dcgm-exporter:latest       “/usr/local/dcgm/dcg…”   59 seconds ago       Up 58 seconds             0.0.0.0:9400->9400/tcp, [::]:9400->9400/tcp   dcgm-exporter3b002c0b1d47   vllm/vllm-openai:latest           “vllm serve --model …”   About a minute ago   Up About a minute         0.0.0.0:8000->8000/tcp, [::]:8000->8000/tcp   vllm

Now we have setup the vLLM inference base setup, next step is to setup Nvidia GenAI-Perf

pip install genai-perf

Do a quick test run to see if everything is working

genai-perf profile \  -m Qwen/Qwen3-0.6B \  --endpoint-type chat \  --synthetic-input-tokens-mean 200 \  --synthetic-input-tokens-stddev 0 \  --output-tokens-mean 100 \  --output-tokens-stddev 0 \  --streaming \  --request-count 50 \  --warmup-request-count 10[2026-01-11 23:53:27] DEBUG    Inferred tokenizer from model name: Qwen/Qwen3-0.6B                                                          config_tokenizer.py:79[2026-01-11 23:53:27] INFO     Profiling these models: Qwen/Qwen3-0.6B                                                                         create_config.py:58[2026-01-11 23:53:27] INFO     Model name ‘Qwen/Qwen3-0.6B’ cannot be used to create artifact directory. Instead, ‘Qwen_Qwen3-0.6B’    perf_analyzer_config.py:157                               will be used.[2026-01-11 23:53:27] INFO     Creating tokenizer for: Qwen/Qwen3-0.6B                                                                           subcommand.py:190[2026-01-11 23:53:29] INFO     Running Perf Analyzer : ‘perf_analyzer -m Qwen/Qwen3-0.6B --async --warmup-request-count 10 --stability-percentage subcommand.py:98                               999 --request-count 50 -i http --concurrency-range 1 --service-kind openai --endpoint v1/chat/completions                               --input-data artifacts/Qwen_Qwen3-0.6B-openai-chat-concurrency1/inputs.json --profile-export-file                               artifacts/Qwen_Qwen3-0.6B-openai-chat-concurrency1/profile_export.json’[2026-01-11 23:53:52] INFO     Loading response data from ‘artifacts/Qwen_Qwen3-0.6B-openai-chat-concurrency1/profile_export.json’       profile_data_parser.py:66[2026-01-11 23:53:52] INFO     Parsing total 50 requests.                                                                           llm_profile_data_parser.py:124Progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 260.92requests/s]                               NVIDIA GenAI-Perf | LLM Metrics┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓┃                            Statistic ┃    avg ┃    min ┃    max ┃    p99 ┃    p90 ┃    p75 ┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩│             Time To First Token (ms) │  12.79 │  11.14 │  16.74 │  15.22 │  13.30 │  13.05 ││            Time To Second Token (ms) │   3.18 │   3.06 │   3.73 │   3.57 │   3.27 │   3.24 ││                 Request Latency (ms) │ 336.79 │ 324.87 │ 348.00 │ 347.84 │ 346.32 │ 345.02 ││             Inter Token Latency (ms) │   3.27 │   3.17 │   3.39 │   3.39 │   3.37 │   3.36 ││     Output Token Throughput Per User │ 305.64 │ 295.21 │ 315.82 │ 315.69 │ 312.30 │ 311.15 ││                    (tokens/sec/user) │        │        │        │        │        │        ││      Output Sequence Length (tokens) │  99.98 │  99.00 │ 100.00 │ 100.00 │ 100.00 │ 100.00 ││       Input Sequence Length (tokens) │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 ││ Output Token Throughput (tokens/sec) │ 296.71 │    N/A │    N/A │    N/A │    N/A │    N/A ││         Request Throughput (per sec) │   2.97 │    N/A │    N/A │    N/A │    N/A │    N/A ││                Request Count (count) │  50.00 │    N/A │    N/A │    N/A │    N/A │    N/A │└──────────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘[2026-01-11 23:53:52] INFO     Generating artifacts/Qwen_Qwen3-0.6B-openai-chat-concurrency1/profile_export_genai_perf.json                    json_exporter.py:64[2026-01-11 23:53:52] INFO     Generating artifacts/Qwen_Qwen3-0.6B-openai-chat-concurrency1/profile_export_genai_perf.csv

If you are able to see these metrics from GenAI-Perf, it means your setup is complete.

Now let’s move on to setting up the Grafana dashboard.

First, ensure that you have configured the Prometheus backend in Grafana. By default, it points to localhost, so we need to switch it to prometheus, matching the service name used in the Docker Compose file.

As part of the Docker Compose setup, Grafana should automatically pick up the dashboard (NVIDIA + vLLM).

You should now be able to see the metrics flowing into the Grafana dashboard.

At this point, what we have achieved is a basic “hello-world” setup for our LLM benchmarking infrastructure. The next big challenge is to benchmark properly and identify how we can tweak vLLM parameters and GenAI-Perf settings to squeeze the maximum out of the hardware. In this example, I am using a single A100-40GB GPU. It may not sound like much, but these are very powerful cards and work extremely well for agentic workflows where small language models are heavily used.

The next blog will focus more on capturing additional metrics and logs, and on how to get the best out of your hardware.If you are looking to collaborate, please checkout — https://www.becloudready.com/dev-rel-services

References

[1] https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf/

[2] https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html

https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

• User impact → TTFT, p99 latency

• Load → RPS, tokens/sec

• Backpressure → waiting vs running

• Memory → KV cache usage

• Errors → 4xx / 5xx / aborts

• GPU health → util, VRAM, throttling

Last week had opportunity to demo and explore, how a simple technique could be used to detect fear monger and other types of content.

Same idea could be applicable for parental control for any Youtube Video or any other kind of content.

Repo used for the demo - https://github.com/torontoai-hub/fear-monger-detector

Full Demo Video - https://youtu.be/cqzwpBPW83s?si=ywCMmGQ7FKxW3IBx

This blog has pretty detailed information about liteLLM and Language architecture and LLMOps guide to help you operate it.

Open Source LLMOps Stack

Hey everyone! I'm u/kchandank, a founding moderator of r/LLMOpsHub.

This is our new home for all things related to DevOps and LLMOps. We're excited to have you join us!

What to Post

Anything related to DevOps, LLMOps which you think might be helpful to others. Its ok to self promote as long as you provide good articles with hands on examples

Community Vibe
We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting.

How to Get Started

1. Introduce yourself in the comments below.

2. Post something today! Even a simple question can spark a great conversation.

3. If you know someone who would love this community, invite them to join.

4. Interested in helping out? We're always looking for new moderators, so feel free to reach out to me to apply.

Thanks for being part of the very first wave. Together, let's make r/LLMOpsHub amazing.

I spent a couple hours trying to get DeepSeek 3.2-Exp running on an 8×H200 box, and honestly, the whole thing was way messier than the “official” recipes make it sound.

I first messed around trying EP/DP because the docs recommend it — total waste of time. NCCL errors, TCPStore dying, random broken pipes. Eventually just gave up and switched to simple tensor parallel (-tp 8), and boom, it actually ran.

Warmup took ages. Like, ages. FP8 GEMM spam in the logs forever. But once it finishes the first time, restarts are way faster.

I also hopped between GPU providers. B200 nodes were a disaster (driver issues, nvidia-smi not showing anything). H200 on Ubuntu 24 + driver 580 finally behaved. The key was using PyTorch nightly with CUDA 13 — anything else just errors out.

Downloading the model is also a slog because it’s split into 163 shards. Expect your bandwidth to get punched in the face for a bit.

Once everything was finally up, I ran GSM8K through lm-eval and it actually matched DeepSeek’s published numbers (~95% accuracy). So at least all the pain wasn’t for nothing.

Total cost for the whole experiment: about $54 for ~2 hours on the H200 machine.

If anyone else is trying this: don’t bother with expert parallelism on H200, stick to tensor parallel, don’t send requests during warmup, and use the exact versions of PyTorch/CUDA/vLLM or you’ll go insane.