Http2 slower then http1 in high volume

[ricardkollcaku]

Im trying to do some benchmarking with spring boot using http1 and http2
and i see i weird behaviour.

Im using for client and server
SPRING BOOT 2.6.6
Java 17
Both using webflux

http2 is bit slower than http1 looks like is not taking advantage of single tcp connection with multiplexing.

you can find the project in https://github.com/ricardkollcaku/http2-test

My Server:
I have enabled http2 in properties :
server.http2.enabled= true
and in using curl

curl -I --http2 http://localhost:8080/sms-request/test
i get:

HTTP/1.1 101 Switching Protocols
connection: upgrade
upgrade: h2c

HTTP/2 200 
content-type: text/plain;charset=UTF-8
content-length: 10

My client is another spring boot 2.6.6
and looks like

 WebClient webClient = WebClient.create().mutate()
       .clientConnector(
            new ReactorClientHttpConnector(
                HttpClient.create().protocol(HttpProtocol.H2C)))
        .baseUrl("http://localhost:8080/sms-request/test").build();

    Flux.range(1, 100000000)
        .concatMap(integer -> webClient.get().retrieve().bodyToMono(String.class))
      .subscribe()
    ;

Using default protocol i can process up to 23,959 requests per second
with H2C protocol i can barely reach 16.000request per second

[violetagg]

@ricardkollcaku We are working on fixes related to the observed behaviour.

[violetagg]

@ricardkollcaku

In the reproducible example you are using flatMap

    Flux.range(1, 100000000)
        .flatMap(integer -> webClient.get().retrieve().bodyToMono(String.class))
        .subscribe()
    ;

While in the comment above you are specifying concatMap

    Flux.range(1, 100000000)
        .concatMap(integer -> webClient.get().retrieve().bodyToMono(String.class))
      .subscribe()
    ;

Please clarify the scenario!

Also with curl you are using connection: upgrade while the WebClient is configured to use H2C prior-knowledge. Please clarify the scenario that you are interested in. Here you can find more about WebClient configuration related to these two scenarios: https://projectreactor.io/docs/netty/release/reference/index.html#_protocol_selection_2

[ricardkollcaku]

Hi @violetagg
The example was just for test (benchmark) propose and i was trying to check how http1, http2 and rsocket compare with each other in terms of speed and i was expecting that with http2 to be faster than http1 but in my benchmark http2 is performing slower than http1.

Yes i have tested it with flatmap and concatmap. i thought maybe there was not working multiplexing with flatmap thats why i tested with concatmap.

But the problem is not flatmap or concatmap is why http2 is performing worst than http1

[violetagg]

@ricardkollcaku a fix is available in 1.0.20-SNAPSHOT. It will be great if you can test it.

[ricardkollcaku]

@violetagg is performing better than preview version but still outperformed by http1
using http1 im able to process up to 23k requests/s
and using http2 im able to process 18K requests/second

and i think http2 should outperform http1 Github:ricardkollcaku/http2-test@f03d780

[kitkars]

Hi @violetagg & @ricardkollcaku

Out of curiosity, I just wanted to try myself the test. Your test looks good. HTTP2 is better than HTTP1.1 when we really use the persistent connection along with its multiplexing capabilities. If you keep creating new connections everytime as part of a request, HTTP2 might not be better and could be worse. Thats what this test seems to confirm.

You also might want to monitor how many TCP connections you are creating by running this.
netstat -tln | grep 8080

If possible and you do not mind, please try this as well once. That is, we create persistent connections with HTTP2 for multiplexing.

private final HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("test")
                                                                            .maxConnections(10)
                                                                            .allocationStrategy(Http2AllocationStrategy.builder().maxConnections(10).build())
                                                                            .build()) ;
                                                                            
WebClient webClient = WebClient.create().mutate()
        .clientConnector(
            new ReactorClientHttpConnector(
                httpClient
            ))
        .baseUrl("http://localhost:8080/sms-request/test").build();                                                                         

In the flatMap - the default parallelism is only 256. Just increase it to 5000

    Flux.range(1, 100000000)
        .flatMap(integer -> webClient.get().retrieve().bodyToMono(String.class), 5000)
        .subscribe()

Now see how many items are getting processed in a second.

Once the test is done, try the same HTTP1.1, the results will not even be close.

My point is - just by switching HTTP1.1 to HTTP2 will not help unless we use the multiplexing.

[ricardkollcaku]

Hi @kitkars
i have tried with your changes (Updated in github https://github.com/ricardkollcaku/http2-test)
Now yes i see that it created just 10 connections but still http1 is outperforming http2 with http1 i process 21K requests/s
with http2 10connections i get around 19k requests/s
i have tried to use different number of connections like 10-20-50 and different parallelism like 254,500,5000 but still http1 is performing better

[kitkars]

@ricardkollcaku @samueldlightfoot

Couple of things to notice.

Issue 1::

I checked your gradle file. As far as I know you never used 1.0.20-SNAPSHOT because (I am a maven user. this comment is based on maven. I believe gradle should also behave more or less same) if you ever want to override the dependency webflux brings, you need to keep that dependency which you want to have , in this case reactor netty 1.0.20-SNAPSHOT, should be on the top of the list. The order of the dependencies matters!! https://stackoverflow.com/questions/31740785/why-order-of-maven-dependencies-matter

Fix: Please fix this by using spring version latest 2.7.1 which brings the latest version of reactor-netty. So you do no longer need 1.0.20-SNAPSHOT anymore.

Issue 2:

We can appreciate the power of reactive programming/webflux etc only when we have more IO intensive tasks compared to CPU intensive tasks. Lets say you have more computation (finding a lot of permutations and combinations, sorting etc) here you will NOT see any major performance improvement just by using reactor. In this case traditional synchronous style code will perform better.
Similarly, to understand HTTP2 multiplexing we really need network involved which causes in transit delay. I feel in this particular test, everything is local and server does not do any processing. So I feel this more CPU intensive. So, I modified your test as shown here.

  @GetMapping("/sms-request/{s}")
  Mono<String> requestResponse(@PathVariable("s") String s) {
    return Mono.just("test :" + s)
            .delayElement(Duration.ofMillis(50))
            .doOnNext(v ->  benchmarkProvider.incrementProcessed());
  }

Additional Info:

HTTP1.1 needs to make new TCP connections for every request. So, if you had noticed you would have created more than 250 TCP connections to get the work done. If you want to improve further, you have to create more connections. I am sure you will hit bottleneck at some point.

HTTP2 on the other hand, you can have few TCP connections. Say 10. But you can get more work done by increasing the flatMap parallelism. Because, it will use same TCP connection to send all the requests. You can not increase the parallelism in HTTP1.1 like that. You can try and you will get connection closed error...etc.

My aim here is not make HTTP2 look good somehow. HTTP2 is different and will perform better only when we have multiplexing. So for multiplexing, increase your parallelism to 10000 or more and see. Whatever HTTP1.1 does with 250 / 500 TCP connections, HTTP2 can do with 1 or 2 TCP connections.

[ricardkollcaku]

@kitkars
Issue 1 ) im using 1.0.20-SNAPSHOT because if you check the dependencies we have now version 1.0.20-snapshot.
anyway i still tried to upgrade spring to version 2.7.1 and the result is the same.
Issue 2) im not trying to compare imperative vs reactive here or to show the power of reactive when we have io, im just trying to compare http1 vs http2 without any delay or other operations just http1 vs http2 with immediate response .
in any case logically http2 should be faster even though we dont have any delay or something as for every request it doesnt need to create new connection.
but what we see in this benchmark (even version 2.7.1) http1 will kill http2 (when no delay) i know is not much a real case scenario but im expecting that even in no delay http2 should be faster
i was expecting that http2 to have a close result with RSocket (slower but close) as both use TCP with multiplexing but the difference is huge. (8 times slower)

Maybe my expectation is wrong but should not http2 beat http1 when it is without delay?

[exper0]

@ricardkollcaku no keep-alive so every single request is establishing new connection, right? That's both for HTTP1 and HTTP2 but it seems HTTP2 is a bit heavier on establishing connection, it's not supposed to be used like that, the whole point of HTTP2 is multiplexing

[Ricard-Kollcaku]

no in http2 im trying to get the benefit of multiplexing. so what i expect is that http2 to be better as it uses the same connection, meanwhile in the result is not better.

[exper0]

@ricardkollcaku so just set connection pool to 1 in both HTTP1 and HTTP2 - even on local machine HTTP2 provides better results. However, if you want meaningful test you should run client and server on different machines to have real TCP connection, not just loopback(localhost). The benefit of HTTP2 is that it may provide same level of concurrency as HTTP 1.1 using much less connections - and connection is actually expensive resource, that's why HTTP2 is better. It just has less overhead on keeping bunch of connections and can provide better throughput
As I can see each connection is being handed by single thread on the server so for the best performance you should have number of connections matched to number of cpu cores on the server. I could even say number of real cores because things like hyper threading makes thing worse on high load