Skip to main content
← ArticlesLire en français
·Alien6 Research

AI in production: the pitfalls of LLM quota management

An experimental study of shared quotas, queues and responses that reach the client too late.

LLMKubernetesArchitectureProduction

A request can consume a resource, produce a correct result and still fail from the client's perspective. The provider has finished its work, but the response arrives after the deadline. This gap between an operation performed and a service delivered becomes critical when several application instances share the same inference capacity.

In the architecture studied by Alien6, Kubernetes pods receive requests, call a shared provider and return its responses. Adding pods increases the application's ability to process requests; it does not necessarily increase the budget available downstream. A control policy can account for that budget correctly while managing waiting time poorly. This is precisely what our measurements show: under quota, the tested global reservation policy sharply reduces the number of responses received on time.

The shared limit behind the replicas

The experiment uses an emulated provider that applies a quota, waits for a defined duration and returns a verifiable result without generating text. This simplification lets us examine pod capacity separately from the capacity of their dependency. A response counts as useful only if it is correct, has HTTP status 200 and reaches the client within two seconds of its scheduled arrival.

Calls to a simulated API

Application instances, quota and timely responses

We compared direct dispatch with reserving quota before the call, under constant-rate Poisson arrivals. Lines connect three-trial means; small dots show each trial.

Direct dispatch, no binding quotaDirect dispatch, shared quotaPrior reservation, shared quota

Useful responses/s (10-second arrival window)

Useful responses per second by application instance countThree series: no binding quota, a quota of 40,000 units per second, and the same quota with global reservation. Large points show the mean of three trials; small points show each observation. The table retains the individual values.

Application instances

At eight instances, we measure 170.73 useful responses/s without a binding quota, 25.47 under quota, and 9.17 with prior reservation.

Read the numerical values
Useful responses divided by the ten-second arrival cohort duration; mean followed by the three individual values for each condition.
Application instancesDirect dispatch, no binding quotaDirect dispatch, shared quotaPrior reservation, shared quota
118.8720.30 / 18.10 / 18.2019.6020.40 / 18.20 / 20.2017.6719.10 / 20.10 / 13.80
239.6341.10 / 36.40 / 41.4025.2024.10 / 25.70 / 25.8014.7013.70 / 14.70 / 15.70
476.2777.60 / 71.80 / 79.4025.1025.20 / 25.40 / 24.7012.9714.80 / 12.30 / 11.80
8170.73175.70 / 169.10 / 167.4025.4726.10 / 24.90 / 25.409.176.90 / 11.00 / 9.60

A useful response is correct, has HTTP status 200 and reaches the client within two seconds of its scheduled arrival. Three repetitions per condition, with an emulated provider: this cohort throughput is not sustained production capacity, and its simulation units are not LLM tokens.

Poisson arrivals at constant intensity; shared 40,000-unit/s quota in the simulationMeasured results and paired comparisons

The blue curve shows behavior without a binding quota or prior reservation: between one and eight pods, mean throughput rises from 18.87 to 170.73 useful responses per second. Each pod provides four active processing slots; while the provider has spare capacity, this additional local capacity allows more requests to be processed. The green curve imposes a shared budget of 40,000 units per second: throughput initially increases, then levels off around 25 useful responses per second. The orange curve adds global reservation; its decline calls for an explanation beyond the provider's ceiling alone.

The approximate plateau follows from the cost of the calls. Each randomly consumes 1,000 or 2,200 units, with equal probability, giving a theoretical mean of 1,600 units. The ratio 40,000 / 1,600 gives approximately 25 calls per second. These are simulation units, not tokens measured on an LLM. The calculation explains why the plateau is near this level without establishing an exact throughput for every experimental window or sustainable production capacity.

This result illustrates a familiar limit of elasticity: adding consumers does not raise the capacity of the resource they share. Pod count alone therefore cannot explain the service delivered; we also need to examine the decisions made before calling the provider.

When reservation makes things worse

Prior reservation addresses a legitimate problem. Several instances must share a quota without spending the same capacity more than once. The experiment's atomic ledger reserves budget before the call, then converts the reservation into consumption without a second debit. That accounting is verified. It does not, however, determine whether an authorized request still has enough time to finish usefully.

Under the tested policy, waiting for a reservation already occupies one of the pod's four active slots. Behind those slots, a bounded queue of 32 requests accumulates waiting time. When budget is unavailable, a request thus holds a processing slot without yet making progress at the provider, while the requests behind it approach their deadlines. The policy has no scheduling mechanism that assesses whether their processing duration fits within the time remaining.

Two observations of the same requests

Quota used and responses received within two seconds

We counted requests that consumed quota in the simulated API and those that produced useful responses in time. We compare eight instances, the same quota and three trials: 5,279 scheduled measurement arrivals per policy.

Correct response received before deadlineQuota used without a useful response before deadline

Direct dispatch

764

764 useful responses; 0 requests that consumed quota without a useful response in time.

Prior reservation

762

275 useful responses; 487 requests that consumed quota without a useful response in time.

Requests that consumed quota, summed across three trials

With prior reservation, 487 of 762 requests (63.9%) consumed quota without a useful response before the deadline.

We use requests consuming quota as the denominator, not all arrivals. This does not mean those requests all started after expiry. We examine the tested policy with four active slots per instance and a bounded queue.

Two-second client deadline; consumed quota remains accounted for after abandonmentTrial records and aggregate analysis

At eight pods, under the same quota of 40,000 units per second, mean useful throughput falls from 25.47 responses per second without reservation to 9.17 with reservation. Both policies receive the same 5,279 scheduled arrivals across three repetitions. Without prior reservation, 764 requests consume quota and produce a useful response. With reservation, 762 consume quota, but only 275 produce a useful response before the deadline.

The other 487 represent 63.9% of the 762 requests that consumed quota, not 63.9% of all arrivals. The number of quota-consuming operations remains similar, while useful responses become much less numerous. This proportion measures neither a share of the units consumed nor the provider's HTTP error rate. It relates actual consumption to the outcome received by the client.

A valid reservation establishes that spending is authorized; it does not guarantee that this spending will still deliver the expected service. Even a correct HTTP 200 response from the provider cannot undo a missed deadline. This does not mean that all 487 calls started after expiry: the deadline can be crossed during their journey. The observed decline concerns this particular organization of waiting, with its occupied slots and lack of a deadline-feasibility check. It does not show that global reservation is always harmful.

What the Qwen experiment adds

A second campaign adds evidence from real Qwen inference, running through Ollama on a CPU to classify HTTP statuses into short JSON labels. The intervention is different: it limits the number of concurrent generation requests to K=1, without an artificially binding consumption quota. When no permit is available, the call is rejected immediately, with no additional queue or retry. The permit remains occupied until successful native completion, even after the client's deadline expires.

The twelve paired comparisons, comprising 24 trials, compare this policy with sending requests without a shared concurrency limit, using one or eight pods and two loads. Each pair receives the same requests at the same times. The cap is respected, but useful responses decrease in eleven pairs and increase in only one. At eight pods under high load, the three repetitions move respectively from 0 to 17 useful responses out of 71 arrivals, from 26 to 14 out of 64, and from 35 to 13 out of 44. The effect changes direction across repetitions.

K=1 is an experimental parameter derived from a calibration rule, with no established optimality or guarantee of meeting the client's ten-second deadline. The policy changes both permitted concurrency and the rejection decision: it does not isolate the effect of coordination. The all twelve paired comparisons remain exploratory. Above all, they show why compliance with a limit must be assessed alongside responses received on time, with rejections included in the denominator.

The architectural lessons

The question extends the history of scheduling under deadlines. As early as 1973, Liu and Layland studied the relationships between capacity, execution time and priorities. Their framework assumes independent periodic tasks, a single processor and preemptive scheduling. It does not directly provide guarantees for distributed inference with variable execution times. It nevertheless reminds us that having a resource available and completing work on time are two distinct requirements.

Contemporary tools address complementary dimensions. Envoy can coordinate rate limits across instances. gRPC can propagate the remaining time between services, depending on the language and its configuration; its cancellation signal still requires the processing code to cooperate in stopping. Ray Serve distinguishes admitted concurrency per replica from the waiting allowed per caller, proxy or handle, and can reject requests beyond a bounded queue. These mechanisms do not, by themselves, guarantee a useful response before the deadline.

None of these tools was compared with our implementation. The demonstrated result concerns a policy whose accounting integrity is preserved while its effectiveness deteriorates. Its architectural implication is specific: admission must be examined alongside where a request waits, the resources it occupies and the time it has left. Checking only quota or concurrency leaves some of the conditions required to deliver the service outside the control policy.

Method and evidence

The emulated campaign comprises 72 trials, with three repetitions per condition; the figures show steady arrival rates, ten-second arrival cohorts and a two-second deadline. Useful throughput tracks scheduled arrivals until their deadlines, then divides the number of useful responses by the cohort duration. The Qwen campaign uses 335-second cohorts and a ten-second deadline; its 24 trials pass the archived data-capture and native-completion checks. A useful response is correct, has HTTP status 200 and is received on time; all scheduled arrivals remain in the denominator. These short trials, or trials limited to CPU classification, with three repetitions per condition, do not establish production sizing.

The protocol, results, validity limits, evidence ledger and complete emulator analysis make it possible to verify the measurements without running another experiment.