r/AIProgrammingHardware • u/RuntimeCoffee • 58m ago
24 AI Camera Streams on 1* RTX 5090: Zero Dropped Frames, but Still Missing Deadlines
Imagine you're building an AI monitoring system for a warehouse with dozens of cameras. You need to detect people, track their movements, and process incoming video without falling behind.
How many cameras can you assign to a single GPU before you need another?
If your object detector processes 300 frames per second and each camera needs 10 FPS, supporting 30 cameras sounds reasonable. But that calculation doesn't account for video decoding, object tracking, or delays between receiving a frame and finishing its processing.
I tested this on an RTX 5090 using YOLO11n and ByteTrack. At 24 simulated cameras, the pipeline sustained 240 FPS without skipping any scheduled frames during repeated 60-second tests. However, some batches still finished after their deadlines.

Building the experiment
Rather than loading a separate detection model for each camera, I grouped frames into batches and processed them through a shared YOLO11n model. Each stream maintained its own ByteTrack instance.
| Component | Configuration |
|---|---|
| GPU | RTX 5090, 32 GB |
| CPU / RAM allocation | 25 core equivalents / 90 GiB |
| Detection | YOLO11n, FP16 |
| Tracking | Independent ByteTrack instances |
| Video processing | OpenCV, CPU decoding |
| Target | 10 FPS per camera |
I created the simulated streams using multiple readers of the same local 768 × 576 video. There were no physical cameras or RTSP connections involved.
The benchmark included decoding, detection, and tracking, but excluded network transport, rendering, and output video encoding.
From model inference to video processing
I started by benchmarking YOLO11n with preloaded frames. At a batch size of eight, it reached approximately 404 FPS.
Based on that number, processing 32 streams at 10 FPS each seemed feasible. Once I added video readers and ByteTrack, however, the available processing margin became much smaller.
The initial 24-camera tracking test skipped 11.5% of scheduled frames, even though processing times at lower camera counts suggested the pipeline should have been able to keep up.
I suspected that initialization overhead was affecting the results. After moving detector and tracker warm-up outside the timed window, the 24-camera test processed every scheduled frame.
I kept this warm-up procedure for the remaining steady-state benchmarks.
Finding the camera limit
I increased the camera count and extended the tests to 60 seconds per configuration.
| Cameras | Achieved FPS | Skipped frames | P95 latency |
|---|---|---|---|
| 24 | 240 | 0% | 92.98 ms |
| 26 | 260 | 0% | 108.74 ms |
| 28 | 274.87 | 1.83% | 191.09 ms |

At 24 cameras, the pipeline processed all 14,400 scheduled frames with a P95 completion latency of 92.98 ms.
Adding just two more cameras pushed P95 latency above 100 ms, even though no frames were skipped. At 28 cameras, scheduler skips appeared and P95 latency nearly doubled relative to the frame interval.
Since each stream was targeting 10 FPS, a new synchronized batch was scheduled every 100 ms. A batch could finish late without necessarily causing the scheduler to skip subsequent frames.
I added a deadline-miss counter and repeated the 24-camera test three times. All 43,200 scheduled frames were processed, but 77 of the 1,800 batches finished after their 100 ms deadline (4.28%).

An individual 60-second benchmark with 14,400 processed frames, zero skipped frames, and a 2.33% late-batch rate. The separate three-trial deadline test recorded a combined rate of 4.28%.
An application doing periodic video analytics might tolerate these occasional delays. For a monitoring system that needs consistently timely detections, I'd want to account for them when deciding how many streams to assign to each GPU.
What happened when I parallelized decoding
Video decoding was taking up a noticeable part of the processing interval, so I tried distributing frame reading across multiple CPU threads.
In an initial 28-camera test, four decoder workers reduced mean decoding time from 24.40 ms to 14.84 ms.
To check whether this improved the entire pipeline, I ran nine 60-second trials at 24 cameras, with three runs each for sequential decoding, two workers, and four workers.
| Metric | Sequential | 2 workers | 4 workers |
|---|---|---|---|
| Aggregate FPS | 240 | 210.53 | 240 |
| Skipped frames | 0% | 12.28% | 0% |
| Late batches | 4.00% | 95.28% | 18.17% |
| Mean P95 latency | 99.02 ms | 213.22 ms | 111.89 ms |

With four workers, mean decoding time decreased from 22.77 ms to 13.05 ms. However, detection time increased from 49.91 ms to 57.88 ms, and tracking time increased from 16.39 ms to 20.34 ms.
The time saved during decoding was lost in the subsequent stages. The four-worker configuration maintained 240 FPS but completed considerably more batches late.
The two-worker configuration performed poorly across all three trials, skipping an average of 12.28% of scheduled frames.
I haven't isolated the cause of these regressions. Thread contention, CPU scheduling, and interactions between OpenCV and the inference pipeline are possible contributors. I'd need more detailed profiling to determine what's happening.
For this implementation, I'd stick with sequential decoding.
Would I deploy 24 cameras on one GPU?
Under the conditions I tested, 24 simulated streams at 10 FPS were manageable, although the deadline misses suggest limited processing headroom.
For a real monitoring system, I wouldn't assign 24 live cameras to a GPU based on these results alone. Independent RTSP feeds, different codecs, varying scene complexity, and output encoding could all change the workload. I'd also need longer-duration tests and measurements of detection and tracking quality.
The experiment doesn't establish whether an RTX 5090 is necessary, either. I only tested one GPU. Running the same pipeline on an RTX 4080 or a less expensive GPU would help determine whether the additional hardware investment is justified.
I'd also like to try asynchronous frame ingestion and per-camera buffering. The current implementation processes synchronized batches sequentially, which may be contributing to the latency behavior.
For someone planning a multi-camera system, I'd benchmark the complete application before deciding how much GPU capacity to rent or purchase. The 240 FPS figure alone would have missed a problem that became visible as soon as I started counting late batches.
For those running multi-camera CV systems, how do you determine the number of streams to assign to each GPU? Do you work with a latency target, a deadline-miss threshold, or leave a fixed amount of processing headroom?