QoSHandler appears incompatible with handlers that intentionally delay response completion

[nitin-karthy]

Summary
We're seeing QoSHandler cause request failures/timeouts when placed in front of a handler that deliberately delays the completion of a response (e.g. a throttling/rate-shaping handler that holds back the final write()/Callback to pace delivery, without blocking a thread — using Scheduler.schedule()).

Setup

• A DelayHandler (our custom implementation) wraps the outgoing Response in a Response.Wrapper that intercepts write(). The last chunk (and therefore the completion callback) is scheduled to fire after a configurable delay via the connector's Scheduler. No worker thread is blocked while waiting.
• Without QoSHandler, this works exactly as expected under concurrent load — every request completes in roughly the configured delay with a 200. But we run the risk of having unbounded queued requests.
• With a QoSHandler wired in front of DelayHandler (limiting concurrent requests via setMaxRequestCount/setMaxSuspend), concurrent requests start failing once load increases.

Suspected root cause
QoSHandler appears to use the Request.addCompletionListener(request, this::resume) as its sole signal that a request has "finished" and a slot can be freed for a queued request. Since our handler intentionally defers that callback until the delayed final write succeeds, QoSHandler treats each request as occupying a slot for the entire delay duration rather than completing once the "real" processing is done.

Question for maintainers
Is there a supported way to compose QoSHandler with handlers that intentionally hold response completion open (for pacing/throttling purposes) without it counting fully against the QoS slot/timeout? Is there a different lifecycle signal we should be using, or is this combination simply not intended to be supported? Any guidance on the right pattern here would be appreciated.

Repro project
A minimal, self-contained Maven project reproducing this (including a parameterized JUnit test that runs the same load with and without QoSHandler to show the behavioral difference) is attached: QoSHandlerRepro. See its README for details on the handler chain, class responsibilities, and how to run/interpret the test.
qoshandlerrepro.zip

[sbordet]

The behavior of QoSHandler is so by design: a request/response processing is not complete until the completion listener fires, and that is the signal to let another request to be processed.

Doing otherwise would overload the system, for example in terms of buffers to hold request/response data, allocated data structures (for example, queued scheduler tasks), and so on.

What is the problem you are trying to solve with your DelayHandler?
I ask because if you detail that, we may discuss alternative solutions.

[nitin-karthy]

Thanks for your response @sbordet .

Background of the DelayHandler: My application is a mock server for APIs. Users of it can simulate different sort of responses for an API endpoint. One of such simulation we support is delaying a response for a few millis (or even seconds - which we highly discourage) for users who wants to test different sort of integration behavior with their applications. Also, this mock server is run as a back-end by our users in high-load (~1000 requests per second) installations to facilitate some perf tests and 3rd party testing services. So performance and efficient use of resources is important to us. Apart from the static delay that is illustrated in the repro project, we also use dynamic delays between a range.

While I totally appreciate the current design of the QoSHandler (clearly to address the constraints you highlighted above), once the requests are processed (correct mock response generated) and the last response chunk delayed, our server could start processing more suspended requests. If only the QoSHandler's current behavior did not limit us!
Welcome any suggestions you may have.

But if you cannot think of any solutions, would it be possible to consider enhancing QoSHandler with some sort of a signalling mechanism (or API) to indicate that the "real-work" on the request is complete and it could resume with the next suspended request? The standard behavior (resume-after-request-completes semantic) must remain as the default the same as-of-today for sure, but the new resume-when-signaled will give us an escape hatch for processing suspended requests ASAP.

[sbordet]

once the requests are processed (correct mock response generated) and the last response chunk delayed, our server could start processing more suspended requests.

It obviously cannot.
In any HTTP protocol version, the client won't be able to send another request until the server has completely finished writing the response, including the last chunk.

It is not a limit of QoSHandler, but rather a limit of the mental model you have of the system: you think your server could start processing more suspended requests, but that's not the case.

It won't happen in a real deployment, and trying to make a benchmark with mocks work in strange ways that are not happening in real deployments would just give you numbers that mean nothing.

When using HTTP/1 with chunks (your reproducer), and HTTP/2 or HTTP/3, delaying the last write means that the client won't see the request to be complete. For HTTP/1, the client won't send the next request on that connection; for HTTP/2-3 it means that no new stream will be opened if the connection is already maxed out wrt the number of streams.

I recommend that you give up this idea of "the real-work is done, even if I won't send the last bytes of the response until later": it is just plain wrong because the "real work" is not done: the last write could be blocked by TCP congestion or flow control, so you have not completed the response, so the "real work" is not complete.

My 2 cents: keep everything as is: your DelayHandler simulates some slow processing, but don't try to take shortcuts introducing some "real work" concept that do not reflect reality in order to try to "speed up" the QoSHandler behavior.