Incorrect pipelining implementations in the tested servers

[remittor]

In a web server, the most difficult thing to implement correctly is pipelining for requests.

Consequences of incorrect implementation:

1. memory consumption without a set limit
2. segmentation fault

To test the indicated problem, I wrote a fairly simple python script: https://gist.github.com/remittor/f2c89f50ff259e6e704fe246ee6c3dd5
Run example test:
$ python3 pl_test.py -h 127.0.0.1 -p 5000 -u /plaintext

---

libreactor

The server ate a lot of memory and did not free it.

---

japronto

The server ate a lot of memory and did not free it.

Server logs:

$ python3 jp_app.py
Accepting connections on http://0.0.0.0:5000
protocol.pause_writing() failed
protocol: <cprotocol.Protocol object at 0x2200d90>
transport: <TCPTransport closed=False reading=True 0x2195d20>
Traceback (most recent call last):
  File "uvloop/handles/basetransport.pyx", line 74, in uvloop.loop.UVBaseTransport._maybe_pause_protocol
AttributeError: 'cprotocol.Protocol' object has no attribute 'pause_writing'

---

vibora

The server ate a lot of memory. After test stop memory frees, but not completely.

Server logs are full of such errors:

Traceback (most recent call last):
  File "vibora/responses/responses.pyx", line 80, in wait_client_consume
AttributeError: 'vibora.protocol.cprotocol.Connection' object has no attribute 'writable'
Task exception was never retrieved
future: <Task finished name='Task-134387' coro=<<coroutine without __name__>()> exception=AttributeError("'vibora.protocol.cprotocol.Connection' object has no attribute 'writable'")>

Traceback (most recent call last):
  File "vibora/protocol/cprotocol.pyx", line 437, in handle_exception
  File "/home/user/wsgi/vibora/build/lib.linux-x86_64-cpython-39/vibora/blueprints.py", line 118, in process_exception
    response = await exception_handler.call(components)
  File "/home/user/wsgi/vibora/build/lib.linux-x86_64-cpython-39/vibora/exceptions.py", line 51, in call
    params[key] = components.get(class_type)
  File "vibora/components/components.pyx", line 96, in vibora.components.components.ComponentsEngine.get
  File "vibora/components/components.pyx", line 77, in vibora.components.components.ComponentsEngine.search_type
ValueError: Vibora can't decide which component do you want, because there at least two types who are a subclass of <class 'Exception'>

---

socketify

There is a segfault, but the author promises to fix it soon.
UPD: Fixed in version 0.0.19

---

And most likely most of the servers in test TFB have problems implementing HTTP pipelining.

[remittor]

drogon

The server ate a lot of memory.

After the test was stopped, the server freed all previously allocated memory.

[franz1981]

I would consider reaching out each framework persona :)
Probably (hopefully) there's some tuning specific per framework e.g. Netty/Vertx/quarkus uses a limit in the read batch size, meaning they won't exceed such limit while reading requests, but are not implementing specific write side backpressure mechanism, unless enforced via specific configs

[fakeshadow]

This is more like general head blocking issue rather than pipeline specific. It would be interesting to see if these frameworks fail the same way with a single large request head. (like garbage filled header value that takes up a lot of memory)

like @franz1981 pointed out in real world web server it's important to use bounded memory for read/write and actively back off when memory buffer is full.

[cirospaciari]

In socketify case was a header parse bug that I introduced (my bad), but i will ship the new version today, wrk + pipelining with lua works flawless, auto cannon also works, something in this script that triggered the right condition.

[remittor]

@fakeshadow

in real world web server it's important to use bounded memory for read/write and actively back off when memory buffer is full.

When I implementing HTTP pipelining in server FastWSGI, I didn’t even think of organizing a separate large storage for incoming packages.
When I receive data from a socket, I simply block further reading from this socket until all the data that was generated as a response to the request is sent to the socket.
The unblocking of reading on the socket will be carried out only after sending responses to all packets from the pipeline.

Therefore, with this implementation, it is not necessary to allocate additional memory, but there is no way to get more than 2 million requests per second in test /plaintext.

[volyrique]

@remittor Have you observed an actual memory leak or just high memory usage? In the latter case I don't really see an issue - requesting memory from the operating system and giving it back has system call overheads, so it makes sense if a server holds onto its allocations and reuses them for future requests. Also, if limiting memory usage is a real concern, cgroups provide a far more reliable way to achieve this, and there is no need to complicate the server code by implementing a similar mechanism there.

[remittor]

The developers of the uvicorn server also stopped reading from the TCP socket until all the responses to the already read batch of requests were sent. Source: https://www.uvicorn.org/server-behavior/#http-pipelining
Only the results of the TFB test do not confirm these words. This is strange.

But by the detailed results of TFB test, you can easily see whether the HTTP server saves memory when reading from the TCP socket of endless requests (pipeline).

Examples from fresh results:

1. fastwsgi
   362,008 / 90,462 = 4.00 (1024 / 256 = 4)
   1,389,417 / 90,462 = 15.35 (4096 / 256 = 16)
2. granian [wsgi]
   354,801 / 89,894 = 3.94 (1024 / 256 = 4)
3. revenj
   363,714 / 90,572 = 4.01 (1024 / 256 = 4)
   1,418,797 / 90,572 = 15.66 (4096 / 256 = 16)
4. tapestry
   353,003 / 89,575 = 3.94 (1024 / 256 = 4)
5. trillium
   347,053 / 90,739 = 3.82 (1024 / 256 = 4)

---

I also noticed this strange situation (when 256 concurrency):

http server	plaintext 256	json 256
fastwsgi	90,462	733,812
granian	89,894	483,045
revenj	90,572	331,850
tapestry	89,575	5,661
trillium	90,739	5,644

I cannot give an explanation for this.