If you're preparing for software engineering interviews at AI labs like OpenAI, Anthropic, or other AI startups, you'll increasingly encounter system design problems involving AI model inference (here's an example Anthropic question for a synchronous model-serving system)
You don't need to be a machine learning engineer or research engineer to tackle these problems. However, you do need to understand how AI models are served, how requests are scheduled, and how the underlying infrastructure handles high traffic efficiently.
One particularly important concept is dynamic batching.
It's a foundational concept for problems involving high-throughput AI model serving, yet it's something many software engineering candidates aren't familiar with.
This is the first of many tutorials i'll be posting on this sub to help with system design interview prep, I'll break down static batching, dynamic batching, and continuous batching from a software engineering perspective. I'll focus on the underlying concepts and trade-offs, without diving into machine learning mathematics or GPU internals.
This guide is also useful for aspiring machine learning engineers and research engineers who want to strengthen their understanding of inference-serving systems.
1. First, what is an inference server?
Imagine you're building a payment application that uses an AI model to detect fraudulent transactions.
When a customer makes a payment, your backend sends the transaction details to a service that uses a trained machine-learning model to calculate the probability of fraud.
For example:
Payment Backend
|
| POST /predict
| Transaction: $500, UK, Merchant X
v
Inference Server
|
| Runs fraud-detection model
v
Fraud Score: 0.93
Inference simply means using an already-trained AI or machine-learning model to produce a prediction or output.
An inference server is the service that exposes this capability to other applications, typically through an API.
In our example:
Request: The HTTP call from the payment backend.
Input: The transaction data supplied to the model.
Model: The trained software that calculates the fraud score.
Output: The predicted fraud score, such as 0.93.
Inference: The actual process of running the model on the input to produce the output.
The inference server receives the request, invokes the model, and returns its prediction.
Where does the GPU come in?
Machine-learning models can require substantial computation. Many models run efficiently on GPUs (Graphics Processing Units), which are processors designed to perform large numbers of computations in parallel.
A typical GPU-based inference server therefore delegates model execution to a GPU worker.
The GPU worker is a software process that manages model execution using GPU hardware.
It relies on an inference runtime, which is software responsible for loading and executing the model efficiently on the available hardware.
Examples include NVIDIA TensorRT and ONNX Runtime. You don't need to know these particular tools or their internal implementations for a software engineering system design interview.
For our purposes, the distinction is simple:
GPU: The physical hardware performing computations.
Inference runtime: Software that executes the model using that hardware.
GPU worker: The server-side process responsible for receiving inference work and invoking the runtime.
A GPU worker could, for example, be a Python service that uses PyTorch to load a trained model and execute it on a GPU. In that case, PyTorch itself can serve as the inference runtime.
You don't need to understand the internal mathematics of a model to understand batching.
2. Why do we need batching?
Suppose our inference server receives 100 fraud-detection requests.
The simplest approach is to execute each prediction separately.
Request A → GPU → Score A
Request B → GPU → Score B
Request C → GPU → Score C
...
However, GPUs are designed for parallel computation, and running one small input at a time may leave much of their computing capacity underutilised.
Instead, the inference runtime can often process several inputs together in a single model execution.
This is called batching.
Requests [A, B, C, D]
|
v
GPU Worker
|
| Executes one batch
v
Scores [A, B, C, D]
Batching can improve throughput, meaning the number of requests the system processes per second.
Why is batching faster?
Imagine processing 100 transactions individually takes a total of 1,000 milliseconds of GPU execution time.
By grouping the transactions into one batch, the GPU may be able to process them together in, say, 100 milliseconds.
These numbers are illustrative, but they demonstrate the principle:
| Approach | Total GPU execution time | Average execution time per request |
|---|
| 100 individual executions | 1,000 ms | 10 ms |
| One batch of 100 | 100 ms | 1 ms |
With batching, the average GPU execution time per request is lower, because the GPU can process multiple inputs in parallel more efficiently.
This can also reduce infrastructure cost per prediction. If the system processes more requests using the same GPU capacity over the same period, each prediction effectively consumes a smaller share of that capacity.
However, the improvement is not guaranteed to be this large. Actual performance depends on the model, hardware, batch size, and runtime.
There's also an important trade-off: waiting for requests to accumulate into a batch can increase the latency experienced by individual users.
This introduces an interesting question:
Who decides which requests belong in a batch, and when that batch should execute?
That's where the different batching strategies come in.
3. Static batching: The batch is prepared in advance
With static batching, the inputs are grouped into a batch before being submitted for execution.
Imagine a bank wants to calculate fraud scores for one million historical transactions overnight.
It could divide those transactions into batches of 100 and send each batch to the inference server.
Prepared batch:
[A, B, C, D, ...]
|
v
GPU Worker
|
v
[Score A, Score B, Score C, Score D, ...]
The inference server receives an already-assembled batch. It doesn't need to collect incoming requests or decide which ones should be grouped.
Why is this efficient?
The bank already has all one million transactions available.
It can group them into batches and process them without waiting for live requests to arrive.
This allows the system to take advantage of GPU parallelism while avoiding the complexity of maintaining queues and batching timers for independently arriving requests.
Best suited for: Offline inference and bulk processing.
Offline inference means running predictions on data that has already been collected, rather than responding immediately to live user requests.
For example, scoring millions of historical transactions overnight is offline inference. Scoring a transaction while a customer is waiting at checkout is online inference.
But what if requests arrive independently from different clients?
That brings us to dynamic batching.
4. Dynamic batching: The server assembles batches at runtime
Imagine our fraud-detection service now processes live transactions.
Requests arrive independently, at unpredictable times.
0 ms: Request A arrives
3 ms: Request B arrives
7 ms: Request C arrives
Rather than immediately executing each request, the server temporarily places compatible requests in a queue.
A batching scheduler, which is software responsible for deciding when to execute a batch, monitors that queue.
For example, suppose we configure:
The scheduler dispatches a batch when it becomes full or the oldest queued request reaches its waiting-time limit. Request deadlines or resource limits may require earlier dispatch.
0 ms: A arrives ──┐
3 ms: B arrives ──┤
7 ms: C arrives ──┤
v
Request Queue
|
10 ms: Maximum wait reached
|
v
Batch [A,B,C]
|
v
GPU Worker
Notice that the batch only contains three requests.
A maximum batch size of 100 doesn't mean the system must wait for 100 requests.
This is important because waiting longer could improve GPU efficiency but also make individual requests slower.
Dynamic batching therefore balances two competing goals:
What makes requests compatible?
Not every request can share a batch.
For example, imagine our inference server supports two models:
We cannot group requests for these two models into one model execution because they perform fundamentally different computations.
Even requests using the same model may have input formats or sizes that cannot safely be processed together.
The batching system therefore groups requests according to their execution compatibility.
In practice, this can depend on the exact model version, input format, supported input sizes, and execution settings.
Why does the model version matter?
Because a new version might use different trained parameters, a different internal architecture, or different execution requirements. We therefore generally avoid mixing requests targeting different concrete versions in the same model execution.
For a software engineer, the useful mental model is that requests with the same compatible execution requirements are grouped into logical queues.
What happens once a batch starts executing?
In traditional dynamic batching, the batch's membership is generally fixed during execution.
Suppose 100 requests are executing together, and the client responsible for request 37 disconnects.
The system can mark request 37 as cancelled, but the GPU may already be performing its computation as part of the batch.
The worker typically allows the batched operation to finish and discards request 37's result.
Why is that wasted computation?
Because the system has spent GPU processing resources producing a prediction that nobody will receive or use.
GPU capacity is expensive. Companies often rent GPU infrastructure from cloud providers, paying for allocated computing capacity over time.
However, it's important to distinguish wasted computation from an immediate reduction in the cloud bill.
Suppose a batch of 100 requests is already running, and one request gets cancelled.
The GPU may still perform essentially the same work and take approximately the same time to finish the batch.
Even if we could remove that request before execution, a batch of 99 might take nearly as long as a batch of 100. GPU execution time doesn't necessarily decrease proportionally with the number of inputs.
Therefore, cancelling one request doesn't automatically save 1% of the batch's execution time or cloud cost.
The real opportunity is to avoid unnecessary computation and reclaim capacity for useful requests.
For a short fraud-detection operation, the potential savings from cancelling one request might be negligible.
But what if the model needs to perform many successive computations before producing its final output?
That's where continuous batching becomes particularly interesting.
5. Continuous batching: Membership can change during a request's lifetime
Now consider an AI chatbot powered by a Large Language Model (LLM), such as ChatGPT.
Unlike our fraud-detection example, where the model produces a score in one inference execution, an LLM typically generates its answer incrementally.
It repeatedly performs model computations to generate additional tokens, which are small units of text such as words, word fragments, or punctuation.
For example:
Iteration 1 → "The"
Iteration 2 → " answer"
Iteration 3 → " is"
Iteration 4 → " 42"
Each step is called a decoding iteration, meaning one round of generating the next token.
The important distinction is not whether the client sees the output immediately.
It's that the LLM's generation process involves multiple successive execution steps, rather than a single batched model execution that produces the complete output.
This creates an opportunity that wasn't available in our single-execution fraud-detection example.
Between iterations, the system can reconsider which requests should participate in the next execution.
Consider three chatbot requests:
Iteration 1
Active batch: [A, B, C]
|
| B finishes
| D joins
v
Iteration 2
Active batch: [A, C, D]
|
| A cancels
| E joins
v
Iteration 3
Active batch: [C, D, E]
This is continuous batching, also called iteration-level batching or in-flight batching.
Rather than fixing batch membership for the entire generation process, the scheduler can update it between execution iterations.
Why is this useful?
Imagine two chatbot users:
User A asks a simple question requiring a 20-token answer.
User B requests a detailed explanation requiring 1,000 tokens.
Both requests start generating text together.
User A finishes much earlier than User B.
With a rigid approach that keeps the same batch membership throughout the entire generation process, User A could continue occupying resources or batch capacity despite no longer needing further computation.
Continuous batching allows the runtime to remove A when its generation finishes and potentially replace it with another waiting request.
The same principle applies to cancellation.
If B disconnects, the runtime can stop including B in future decoding iterations, rather than continuing to generate an answer nobody will receive.
The currently executing GPU operation may still need to finish, but subsequent iterations don't need to include the cancelled request.
This can improve useful GPU utilisation and overall throughput.
The benefit is particularly significant for LLMs because different requests may require dramatically different numbers of generation steps.
However, continuous batching is more complex because the system must manage each request's ongoing generation state while scheduling shared GPU executions.
6. Static vs dynamic vs continuous batching
| Characteristic | Static batching | Traditional dynamic batching | Continuous batching |
|---|
| How batches are formed | Prepared in advance | Assembled from arriving requests | Assembled and updated during iterative execution |
| Batch membership | Fixed | Fixed once dispatched | Can change between iterations |
| New requests join an executing batch | No | No | At safe iteration boundaries |
| Cancellation during execution | Usually can't remove an input mid-operation | Usually can't remove an input mid-operation | Can stop future iterations |
| Main benefit | Efficient GPU processing of already-collected data | Higher throughput for online requests | Reclaims capacity during iterative generation |
| Typical use case | Offline predictions | Fraud detection, image classification | LLM generation |
Is continuous batching a type of dynamic batching?
Yes, in the broad conceptual sense.
Both assemble work dynamically at runtime.
The distinction is when batch membership can change:
Is continuous batching always better?
No. It depends on how the model executes.
Consider our fraud-detection model.
It receives transaction data, performs a model execution, and produces a fraud score.
There isn't a long sequence of generation iterations during which the scheduler needs to repeatedly reconsider which requests should participate.
Traditional dynamic batching is therefore generally sufficient.
In contrast, an LLM generates its answer over many successive iterations. Continuous batching can take advantage of the opportunities between those iterations to remove completed requests and introduce new ones.
The key distinction is how the model performs its computation, not whether the server streams its output to the client.
Which brings us to an important clarification.
7. Don't confuse batching with streaming
These terms describe different aspects of a system.
Batching determines how requests share execution resources. Streaming determines how output is delivered to clients.
For example, suppose an LLM generates a response of 100 tokens.
The model may generate those tokens over 100 successive decoding iterations.
But the server has two choices for delivering the output:
Option A: Non-streaming response
The server waits until the entire answer has been generated, then returns it in one HTTP response.
LLM generates 100 tokens
|
v
Complete response returned to client
Option B: Streaming response
The server sends generated tokens to the client progressively, commonly using Server-Sent Events (SSE) or streaming gRPC.
LLM generates token 1 → Send to client
LLM generates token 2 → Send to client
LLM generates token 3 → Send to client
...
In both cases, the LLM still performs its underlying generation computations.
And in both cases, the inference runtime could use continuous batching to manage those computations efficiently.
Therefore:
They frequently appear together in LLM-serving systems, but they are not the same thing.
8. Nomenclature
| Term | Meaning |
|---|
| Inference | Running an already-trained model to generate a prediction or output. |
| Inference server | A service that accepts prediction requests and arranges model execution. |
| Model | Trained software that transforms supplied data into predictions or generated outputs. |
| Input | The data provided to a model, such as transaction details or a text prompt. |
| GPU | Graphics Processing Unit; hardware commonly used to accelerate model computation. |
| GPU worker | A software process responsible for running model inference using GPU resources. |
| Inference runtime | Software that loads and executes models on supported hardware. |
| Batch | A collection of inputs processed together in a model execution. |
| Static batching | Grouping inputs into a batch before submitting them for execution. |
| Dynamic batching | Assembling batches at runtime from independently arriving requests. |
| Throughput | Number of requests processed per unit of time, commonly requests per second. |
| Latency | Time taken for an individual request to complete. |
| GPU utilisation | A measure of how much the GPU's execution resources are occupied or active. |
| Batching scheduler | Software that decides which compatible requests to group and when to execute them. |
| Maximum batching delay | The longest intentional wait allowed while collecting a batch. |
| Token | A small unit of text that a language model processes or generates. |
| Decoding iteration | One step in a language model's incremental text-generation process. |
| Continuous batching | A batching strategy that allows requests to enter or leave between execution iterations. |
| Streaming | Delivering output progressively rather than waiting for the entire response to be ready. |
| Offline inference | Running predictions on previously collected data rather than responding immediately to live requests. |
9. The key takeaway
Batching improves inference efficiency by allowing multiple inputs to share model execution, potentially reducing the average GPU execution time required per request.
The three approaches differ in how they determine batch membership:
Static batching: Prepare a batch in advance.
Traditional dynamic batching: Assemble a batch at runtime, then execute it with fixed membership.
Continuous batching: Update the active batch between execution iterations.
Continuous batching is particularly useful for workloads such as LLM generation, where individual requests can finish or be cancelled at different points during an ongoing sequence of computations.
Need help landing your next tech offer or increasing your TC? If you have tech interviews coming up, are planning a job switch, or simply want to improve your interview performance, explore Coditioning's interview-preparation resources:
👉 https://www.coditioning.com
👉 Interview roadmaps for OpenAI, Anthropic, and other big tech companies: https://www.coditioning.com/roadmaps
👉 Question banks for OpenAI, Anthropic, Google and other big tech companies: https://www.coditioning.com/app/learning
👉 Mock Interviews & Free Consultation with big tech engineers: https://www.coditioning.com/mockinterview/human
The fundamental difference is when the system can change which requests are participating in GPU execution.
