Calibrated Request Routing Improves Disaggregated LLM Serving Goodput by Only 3%

It is a request router that estimates the additional completion time on each serving instance using exact prompt length, predicted output length, post-admission KV cache pressure, and SLO class, then routes to the instance with the lowest predicted cost.

The gains are marginal (~3% goodput) and vanish in small pools, so the operational complexity may not be justified unless you operate large, heterogeneous inference pools.

Those systems separate compute-heavy prefill from memory-heavy decode on distinct GPU pools; this study focuses on the routing layer that decides which instance handles each request after separation.

Hardware calibration is critical: using simulator-derived constants costs 4.5 goodput points and about 40% of the tail latency advantage, reducing the scorer to little more than queue counting.