When 503s Are the Right Answer (Part 2 of LLM Inference Bottlenecks)

In Post 1, I doubled the request arrival rate on an Azure-derived replay and diagnosed what limit vLLM was hitting first.¹

Before pushing harder, I needed a definition of what counted as a request being served well.

I used time to first token (TTFT), the time a user waits before the response starts streaming. At low load, p99 TTFT was 689 ms. I used that as an experimental SLO: a request counted as SLO-compliant if its first token arrived within 689 ms.

Using p99 meant almost every request in the healthy run still counted as good. I wanted the threshold to pick up latency from load. That 689 ms is specific to this workload, not a universal latency target.

A client sent requests at their recorded arrival times, without waiting for earlier requests to finish. Here’s what the two arrival rates looked like:

	~2.5 req/s	~5.1 req/s
output tok/s	277	506
requests/s meeting SLO	2.51	1.01
TTFT p50	142 ms	13.1 s
completed / offered	1,533 / 1,533	3,065 / 3,065

Output throughput rose 83%. Every request completed in both runs. By output throughput, the heavier run looked better. By completion rate, nothing looked wrong.

But SLO-compliant request throughput fell about 60%, from 2.51 to 1.01 req/s. The server was completing everything, but fewer requests were getting their first token on time.

What I cared about was how many requests per second still met the latency target. I used goodput to measure that.

Goodput

For this replay, I computed goodput as:

completions meeting SLO / 600 seconds

Requests arrived over 600 seconds, but the overloaded server kept draining for another ~80 seconds. I used those 600 seconds as the denominator for every run, regardless of how long the server took to finish the backlog. Only completions that meet the 689 ms TTFT target count toward goodput. Rejected requests count as zero.

Admission Control

Post 1 showed requests piling up in vLLM’s waiting queue. By the time many of them started streaming, they had already missed the latency target. What if I stopped some of that work from entering the queue?

I built a Go gateway in front of vLLM. It counted requests that had been forwarded to vLLM but hadn’t finished yet. Once that count reached M, new arrivals got a 503 instead of being forwarded.²

I tried three limits on the higher-rate workload: M=8, M=32, and M=64.

arm	goodput (req/s)	TTFT p50	completed	rejected (503)
ungated	1.01	13.1 s	3,065	0
M=8	2.87	173 ms	1,766	1,299
M=32	4.04	305 ms	2,833	232
M=64	3.25	473 ms	2,935	130

Of the three limits I tried, M=32 worked best. Goodput rose from 1.01 req/s with no gateway to 4.04 req/s, while median TTFT fell from 13 seconds to 305 ms.

It wasn’t just getting a better score by rejecting requests. Without the gateway, about 600 requests met the latency target. With M=32, 2,421 did, even though 232 requests were rejected before reaching vLLM.

M=8 went too far: nearly half or 1,299 requests got 503s. M=64 went the other direction and let enough requests through that TTFT started to rise again.

Doesn’t vLLM already have `max_num_seqs`?

M=32 worked. But vLLM already has max_num_seqs, which limits how many sequences the scheduler processes in a single iteration. The default is 256.

Why do I need a gateway at all? Why not set max_num_seqs=32?

What `max_num_seqs=32` produced

The gateway M=32 experiment is the same one from the table above. The only change was replacing the gateway with max_num_seqs=32.

	gateway M=32	`max_num_seqs=32`
goodput (req/s)	4.04	0.56
TTFT p50	305 ms	8,823 ms
completed	2,833	3,065
rejected (503)	232	0
SLO attainment (of completed)	85.5%	10.9%

The engine completed every request with no rejections and no failures, so by completion rate, it looked fine.

Goodput told a different story: 0.56 req/s, worse than the ungated 1.01 req/s.

Of the 3,065 completions, only 334 met the latency target. Nearly nine out of ten completed too late to count as goodput.

Why max_num_seqs=32 isn’t the same as M=32

max_num_seqs=32 limits how many sequences the scheduler processes per iteration, but the server still accepts every request that arrives. They enter the internal waiting queue. vLLM processes them eventually, so the completion rate looks fine. But by the time a request gets its first token after seconds of waiting, the SLO may already have been breached.

max_num_seqs also doesn’t limit the backlog. It limits how many sequences vLLM can process in an iteration. The rest wait in the queue.

The gateway works differently. When 32 forwarded requests haven’t completed yet, the next arrival gets a 503 and never reaches vLLM. It never joins vLLM’s internal waiting queue. With the gateway, at most 32 requests could be inside vLLM and unfinished at once.

vLLM had just merged admission control

While writing up these results, I found that vLLM had just merged --max-num-queued-reqs, which adds a native admission limit that works on the same basic idea as my gateway: cap unfinished requests and reject excess arrivals.

This wasn’t a new technique I had come up with. The PR author says their production setup already relied on a load balancer to pre-admit requests that could meet its QoS targets, and described the change as porting something they had been using in production for a long time. The new flag moved a simple version of that control into vLLM itself.

At the time of this experiment, the feature hadn’t landed in a stable release yet, but I could test the commit that introduced it.

If the gateway worked because it capped unfinished requests, vLLM’s new admission limit should show the same pattern.

I ran four experiments on one server with the same replay requests and GPU:³

experiment	goodput (req/s)	TTFT p50	completed	rejected (503)
ungated	3.59	408 ms	3,065	0
`max_num_seqs=32`	1.79	1,318 ms	3,065	0
gateway M=32	4.35	258 ms	2,903	162
`max_num_queued_reqs=32`	4.35	254 ms	2,897	168

The two admission limits behaved almost the same: TTFT p50 around 255 ms, goodput around 4.35 req/s, and no more than 32 unfinished requests at any time.

max_num_seqs=32 controlled execution, so requests could still enter and wait. Gateway M=32 and native max_num_queued_reqs=32 controlled admission, so excess requests were rejected before they could join the queue.

It was gratifying to see vLLM land its own native version of this admission control just as I was writing up my experiment results. So of course I had to try it too. It was so cool to see my gateway behave almost exactly like the implementation that had just landed in vLLM.

Same hardware and serving configuration as Post 1: Qwen2.5-7B-Instruct, BF16, A100-PCIE-40GB, vLLM 0.27.1. The opening comparison uses two replays of the same Azure trace at different rates. The gateway and max_num_seqs=32 experiments used the same replay on the same GPU, TTFT <= 689 ms and the same 600-second arrival window. ↩
vLLM 0.27.1 had no built-in admission limit based on queue depth or unfinished requests. max_num_seqs limits how many sequences vLLM can process in one iteration and not how many requests the server accepts, and the internal waiting queue was unbounded. ↩
Based on vLLM commit dafbef15, the pull request that introduced --max-num-queued-reqs. Same GPU model (A100 PCIe 40GB), same replay. All four experiments ran on one GPU. The ungated baseline shifted substantially between versions (3.59 vs 1.01 req/s goodput), so the absolute values should not be compared with the v0.27.1 results earlier in this post. Repeats of the admission experiments stayed in the 4.32–4.35 req/s range. ↩

← Writing