How to handle HTTP/2 GOAWAY with HttpClient?

Issue

I am trying to continuously send GET and POST requests to a REST API every few minutes. The issue is that after exactly 1000 requests I receive a GOAWAY frame (and an IOException):

The GOAWAY frame (type=0x7) is used to initiate shutdown of a connection or to signal serious error conditions.
_{§ 6.8, RFC 7540}

---

I did a fair bit of research and found that not only is 1000 requests nginx's default maximum, Cloudfront (related Chromium issue) and Discord also exhibit the same behavior.

I tried to reproduce this problem with a local nginx server with the default HTTP/2 configuration:

server {
    listen 443 http2 ssl;
    http2_max_requests 1000;
    ...
}

var client = HttpClient.newBuilder()
        .version(HttpClient.Version.HTTP_2)
        .build();

for (var i = 0; i < 1100; i++) {
    var url = URI.create(String.format("https://localhost/images/test%d.jpg", i));

    var request = HttpRequest.newBuilder().uri(url).build();

    client.send(request, HttpResponse.BodyHandlers.discarding());
    System.out.printf("Image %d processed%n", i);
}

And after approximately 1000 requests, I get a GOAWAY error as expected:

...
Image 998 processed
Exception in thread "main" java.io.IOException: /127.0.0.1:49259: GOAWAY received

---

My first thought would be to check if the exception message contains the string "GOAWAY" and then retry the request accordingly:

try {
    client.send(request, HttpResponse.BodyHandlers.discarding());
} catch (IOException e) {
    if (e.getMessage().contains("GOAWAY")) {
        client.send(request, HttpResponse.BodyHandlers.discarding());
    } else throw e;
}

My issue with this approach is that the string comparison seems like it may be fragile. Additionally, since all I have is an IOException with a message, I can't differentiate between GOAWAY frames with a genuine error code (in which case I should probably stop sending requests) and those with NO_ERROR (in which case I could probably retry the request).

How should I correctly deal with/handle GOAWAY errors (apart from using HTTP/1.1 instead)?

Solution

A server is entitled to close connections at any time, for any reason.

In the HTTP/2 GOAWAY frame there is the indication of what was the last stream processed by the server, so the client can know what stream needs to be resent when a connection is closed.

Unfortunately, the lastStreamId is not surfaced in java.net.http.HttpClient, so there is no way to know it and take appropriate actions.

Your alternative could be to use other clients that support surfacing the lastStreamId, or use a lower level HTTP/2 client where you will have the GOAWAY frame available and therefore access to the lastStreamId.

[Disclaimer, I am the Jetty HTTP/2 implementer]
Jetty supports a lower level HTTP/2 client that you can use for your use case - you may want to give it a try. You can find an example of how to use Jetty's HTTP2Client here.

Answered By - sbordet