LLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability requires a different metric set than typical web application monitoring, because request duration alone hides the two numbers that actually determine user experience: time-to-first-token (TTFT) and inter-token latency (ITL, sometimes called time-per-output-token). A request that takes 8 seconds total could be a fast 200ms TTFT with slow token-by-token streaming, or a slow 6-second TTFT with fast generation after that, and users experience those very differently even though total latency looks identical on a naive dashboard. Add throughput and GPU utilization tracking, and you have the four-metric core that catches capacity problems before users file a ticket.
TTFT: Time to First Token
TTFT measures the delay from request submission to the first generated token appearing, which is dominated by the prefill phase, queue wait time, and prompt length. TTFT is the metric users feel most directly in a chat interface, since a slow TTFT reads as the application hanging even if generation is fast once it starts. TTFT degrades nonlinearly under load: it stays flat as concurrency rises until queue depth hits a threshold, then climbs sharply, which makes it a leading indicator of approaching capacity limits well before aggregate throughput saturates. Track TTFT at p50, p95, and p99, not just average, since the p99 tail is where user complaints originate and averages hide it completely.
- Track p50, p95, p99 separately; averages hide the tail latency users actually complain about
- TTFT degrades nonlinearly with concurrency, making it a leading indicator of capacity problems
- Long prompts and cache misses both inflate TTFT; separate these causes in your dashboard if traffic mixes both
- Alert on relative TTFT increase (e.g., p95 up 50 percent week over week), not just an absolute threshold
Inter-Token Latency and Generation Throughput
ITL measures the time between successive generated tokens once streaming starts, and it determines whether streamed output feels smooth or stutters. ITL is primarily driven by how much compute each decode step costs, which rises with batch size (more concurrent sequences competing for the same forward pass) and with model size. Aggregate throughput, typically measured in tokens per second across all concurrent requests, is the metric that determines your effective serving capacity and cost per token, but a high aggregate throughput number can coexist with a poor per-user experience if --max-num-seqs is tuned too aggressively toward throughput at the expense of individual request latency. Track both together: throughput tells you what the GPU is capable of, ITL tells you what any individual user actually experiences at your current concurrency setting.
GPU Utilization Dashboards That Predict Problems
Raw GPU utilization percentage (from nvidia-smi or DCGM) is necessary but insufficient on its own, since a GPU can show 95 percent utilization while KV cache is nearly exhausted and about to trigger preemption, a very different situation from 95 percent utilization with headroom. Build dashboards tracking GPU compute utilization, GPU memory utilization, and KV cache utilization as three separate lines, since they diverge meaningfully once prefix caching and PagedAttention are active. Add queue depth (requests waiting for an available batch slot) as a leading indicator, since queue depth rising while GPU utilization is already near ceiling is the clearest early signal that you need to scale out before TTFT visibly degrades. NVIDIA DCGM Exporter feeding Prometheus and Grafana is the standard stack for GPU-level metrics in 2026 self-hosted deployments.
- GPU compute utilization, memory utilization, and KV cache utilization as three separate tracked lines
- Queue depth as a leading indicator, alerting before TTFT visibly degrades for users
- DCGM Exporter plus Prometheus and Grafana is the standard stack for GPU-level metrics
- Preemption/swap events (sequences evicted from KV cache under memory pressure) should be counted and alerted on directly
Building the Dashboard and Alert Thresholds That Matter
A production LLM observability dashboard needs four panels minimum: TTFT (p50/p95/p99), ITL (p50/p95/p99), aggregate throughput (tokens/sec), and the three-way GPU utilization breakdown with queue depth. Add request volume and error rate as standard application metrics, plus prefix cache hit rate if enabled, since a dropping hit rate silently erodes the TTFT benefit you built prompt structure around. Set alerts on relative change and on leading indicators (queue depth, KV cache utilization approaching ceiling) rather than only on absolute latency thresholds breached, since absolute thresholds catch problems only after users are already affected, while leading indicators give you time to scale before that happens.
How Netray Builds Observability Into On-Prem Deployments
Netray delivers a Grafana dashboard covering TTFT, ITL, throughput, and the three-way GPU utilization breakdown as a standard artifact on every on-prem inference deployment, wired to DCGM Exporter and vLLM's native Prometheus metrics endpoint. We set alert thresholds against the client's actual measured baseline rather than generic defaults, and we specifically instrument queue depth and KV cache utilization as leading indicators so scaling decisions happen before users notice degraded response times. For clients running regulated workloads, this observability stack runs entirely on-premises alongside the inference servers themselves, with no metrics data leaving the network boundary.
Frequently Asked Questions
What is the difference between TTFT and inter-token latency?
TTFT (time-to-first-token) measures delay from request submission to the first generated token appearing, dominated by prefill and queue wait time, and is the metric users feel most directly as application responsiveness. ITL (inter-token latency, or time-per-output-token) measures the gap between successive tokens once streaming starts, determining whether output feels smooth or stutters. A request can have fast TTFT but slow ITL or vice versa, and total latency alone hides which one is the actual problem.
What GPU metrics matter most for LLM serving observability?
Track GPU compute utilization, GPU memory utilization, and KV cache utilization as three separate lines rather than one aggregate number, since they diverge once PagedAttention and prefix caching are active. Add queue depth as a leading indicator: rising queue depth while GPU utilization is near ceiling signals you need to scale before TTFT visibly degrades for users. DCGM Exporter feeding Prometheus and Grafana is the standard stack for capturing these in a self-hosted deployment.
Should I alert on average latency or percentile latency for LLM serving?
Percentile latency, specifically p95 and p99, not average. Averages hide the tail latency that generates user complaints, since a handful of severely slow requests can be masked by many fast ones in an average calculation. Track TTFT and ITL at p50, p95, and p99 separately, and alert on relative change in the p95/p99 values, since that catches degradation trends before they cross an absolute threshold and become visible to users.
Key Takeaways
- 1TTFT: Time to First Token: TTFT measures the delay from request submission to the first generated token appearing, which is dominated by the prefill phase, queue wait time, and prompt length. TTFT is the metric users feel most directly in a chat interface, since a slow TTFT reads as the application hanging even if generation is fast once it starts.
- 2Inter-Token Latency and Generation Throughput: ITL measures the time between successive generated tokens once streaming starts, and it determines whether streamed output feels smooth or stutters. ITL is primarily driven by how much compute each decode step costs, which rises with batch size (more concurrent sequences competing for the same forward pass) and with model size.
- 3GPU Utilization Dashboards That Predict Problems: Raw GPU utilization percentage (from nvidia-smi or DCGM) is necessary but insufficient on its own, since a GPU can show 95 percent utilization while KV cache is nearly exhausted and about to trigger preemption, a very different situation from 95 percent utilization with headroom. Build dashboards tracking GPU compute utilization, GPU memory utilization, and KV cache utilization as three separate lines, since they diverge meaningfully once prefix caching and PagedAttention are active.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
AI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolLLM Inference Cost Per Million Tokens Calculator
Convert GPU hourly cost, cluster throughput, and real-world utilization into a defensible self-hosted cost per million tokens, exposing what idle GPU time actually costs you.
Terms used in this article
Need production-grade observability for your self-hosted LLM deployment? Netray will build the dashboard and alert thresholds tuned to your real traffic before your first capacity incident.
Related Resources
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
AI & AutomationLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
AI & AutomationKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.