The Serving Challenge

Training a model is just the beginning. Serving it to millions of users is an infrastructure challenge.

Key Metrics:

• Throughput: Requests per second (RPS).
• Latency: Time to first token (TTFT) or total generation time.
• Cost: GPU hours are expensive ($2-$4/hr for A100).

1. Serving Patterns

Real-Time (Online) Inference

• SLA: Low latency (< 200ms).
• Architecture: REST/gRPC API.
• Example: ChatGPT, Fraud Detection during checkout.
• Challenges: Auto-scaling GPUs based on traffic.

Batch Inference

• SLA: High latency (Hours/Days).
• Architecture: Cron Job / Airflow DAG.
• Example: Nightly product recommendation generation.
• Pros: High throughput, efficient GPU usage (packed batches).

Streaming Inference

• SLA: Instant Feedback.
• Architecture: Server-Sent Events (SSE) or WebSocket.
• Example: LLM token generation (typing effect).

2. Optimization Techniques

How to make models faster and cheaper?

Dynamic Batching

Process multiple requests in parallel on the GPU.

• Without Batching: 1 req = 10ms compute + 10ms memory overhead. Total = 20ms.
• With Batching (Size 8): 8 reqs = 15ms compute + 10ms memory. Total = 25ms. Per req = 3ms!

Tools: NVIDIA Triton Inference Server, TorchServe, vLLM.

Quantization (Shrinking Models)

FP32 (32-bit float) is standard but huge.

• FP16: Half precision. 2x speedup, 50% RAM. Minimal accuracy loss. Most common.
• INT8: 8-bit integer. 4x speedup, 25% RAM. Needs calibration.
• GPTQ / AWQ: 4-bit quantization for LLMs. Runs Llama-3-70B on 2x A100s instead of 4x.

KV Caching (For LLMs)

Transformers recompute the entire sequence for every token. KV Cache saves the Key/Value matrices of past tokens in GPU VRAM so we only compute the new token.

• Tradeoff: Increases VRAM usage significantly.

3. Deployment Infrastructure

Model Registry

Version control for weights (model.pt).

• Tools: MLflow, HuggingFace Hub, AWS S3.
• Blue/Green Deployment: Test model v2 on 1% traffic before full rollout.

Serverless Inference

• AWS Lambda: Good for CPU models (XGBoost, small BERT). Bad for big GPUs (Cold start > 10s).
• Specialized: Modal, RunPod, Replicate. Spin up containers only when requests arrive.

Code Example: Triton Client

NVIDIA Triton handles batching and scheduling automatically.

import tritonclient.http as httpclient
import numpy as np

# 1. Connect to Inference Server
client = httpclient.InferenceServerClient(url="localhost:8000")

# 2. Prepare Input (Batch Size 1)
input_data = np.array([[0.5, 0.2, 0.1]], dtype=np.float32)
inputs = [
    httpclient.InferInput("input_node", input_data.shape, "FP32")
]
inputs[0].set_data_from_numpy(input_data)

# 3. Request Prediction
results = client.infer(model_name="my_model", inputs=inputs)

# 4. Get Output
output_data = results.as_numpy("output_node")
print(f"Prediction: {output_data}")

4. LLM Serving: Where the Rules Change

Classic model serving assumes fixed-size inputs and outputs — one request, one forward pass, ~10ms. Autoregressive LLMs break every one of those assumptions, which is why a specialized serving stack (vLLM, TensorRT-LLM, SGLang) emerged.

Continuous Batching

Naive ("static") batching waits for 8 requests, runs them together, and returns when all finish. But generation lengths vary wildly — one user asks for a haiku, another for an essay. The haiku's GPU slot sits idle while the essay grinds on.

Continuous batching operates at the token level: after every generation step, finished sequences exit the batch and queued requests join immediately. GPU utilization jumps dramatically — vLLM reported over 10x throughput versus naive serving largely from this scheduling change.

PagedAttention

KV caches were traditionally allocated as contiguous max-length buffers: a 2,048-token reservation for a request that generates 50 tokens wastes ~97% of the space. PagedAttention (vLLM's core idea) manages KV memory like an OS manages RAM — small pages allocated on demand, with page tables mapping logical sequence positions to physical blocks. Memory waste drops to a few percent, which converts directly into larger batches and more throughput per GPU.

The Two Latency Numbers

LLM UX is governed by two separate metrics, tuned differently:

• Time to First Token (TTFT): dominated by prompt processing ("prefill") — compute-bound, benefits from tensor parallelism and prompt caching.
• Time Per Output Token (TPOT): the typing speed — memory-bandwidth-bound, benefits from quantization and speculative decoding (a small draft model proposes tokens; the big model verifies several at once).

A chatbot with 5s TTFT feels broken even at fast TPOT; a batch summarizer doesn't care about TTFT at all. Know which one your product needs before choosing hardware.

5. Reliability Patterns for Inference Fleets

GPU services fail in ways stateless web services don't, and the standard playbook adapts:

• Request hedging with care: retrying a slow LLM request on a second replica doubles GPU cost — expensive hedging. Prefer deadline-aware scheduling and backpressure: reject early at the queue when the fleet is saturated rather than letting latency balloon.
• Health checks that exercise the GPU: a process can respond to HTTP while its CUDA context is wedged. Real readiness probes run a tiny inference.
• Warm pools over cold starts: loading a 140GB model across 8 GPUs takes minutes. Autoscaling must be predictive (scale on queue depth trends, keep N warm spares), and rollouts must be gradual — you cannot blue-green an entire GPU fleet instantly without doubling a very large bill. See Blue-Green Deployment.
• Shadow testing for quality: model v2 can be faster and worse. Route a copy of live traffic to the candidate, log outputs, and evaluate offline before shifting real users — accuracy regressions don't show up in latency dashboards.

Interview Tips 💡

• "CPU vs GPU?" — GPU for deep learning (Matrix Mult parallelization). CPU for decision trees (XGBoost) or very small models.
• "Continuous batching vs dynamic batching?" — Dynamic batching groups whole requests; continuous batching admits/evicts at token granularity. For LLMs, the latter is the throughput unlock.
• "How would you cut LLM serving costs in half?" — Quantization (FP8/INT4), prompt/prefix caching for shared system prompts, speculative decoding, and routing easy queries to a smaller model (cascade serving).
• "What is Cold Start?" — Loading a 10GB model from disk to GPU memory takes 10-20 seconds. Severely impacts serverless latency.
• "Multi-Model Serving" — Loading 10 different models on 1 GPU. If VRAM fills up, swapping to CPU RAM kills performance. Use LoRA adapters to share base weights.

Related Concepts

• Container Architecture
• Load Balancing
• RAG Architecture