Round 22 Completed

[NateBrady23]

Hi everyone!

Round 22 results are in: https://tfb-status.techempower.com/results/66d86090-b6d0-46b3-9752-5aa4913b2e33

We could use some help from the community to verify these results are aligned with what is expected. (This doesn't mean your particular framework performed better/worse than expected.)

Also, if you'd like to be included in the Round 22 blog post, feel free to write a paragraph about the work you did to improve your results from round to round or a link to your blog post that speaks to those changes. You can leave that here or email them to me at [email protected].

[NateBrady23]

Results look good. We will be posting them this week. Still looking for some write ups!

[franz1981]

@nbrady-techempower I missed this :( I still have time to send a wrap-up for me (adding @vietj in cc too) re Netty/Vertx/Quarkus?

[NateBrady23]

@franz1981 Yes, I got yours. Thanks for sending that over!

[fundon]

List a few of Rust's performance optimizations.

In a real production environment, several approaches can be tried to optimize the application:

1. Specify memory allocators

   • mimalloc

   • snmalloc-rs

   • jemallocator

2. Declaring static variables

   • once_cell

   • lazy_static

3. Putting a small portion of data on the stack

   • smallvec

   • tinyvec

4. Using a capacity to new vector or hash, a least capacity elements without reallocating

5. SIMD

[fakeshadow]

List a few of Rust's performance optimizations.

In a real production environment, several approaches can be tried to optimize the application:

1. Specify **memory allocators**
   
   * [`mimalloc`](https://crates.io/crates/mimalloc)
   * [`snmalloc-rs`](https://crates.io/crates/snmalloc-rs)
   * [`jemallocator`](https://crates.io/crates/tikv-jemallocator)

2. Declaring **static variables**
   
   * [`once_cell`](https://crates.io/crates/once_cell)
   * [`lazy_static`](https://crates.io/crates/lazy_static)

3. Putting a small portion of data on the **stack**
   
   * [`smallvec`](https://crates.io/crates/smallvec)
   * [`tinyvec`](https://crates.io/crates/tinyvec)

4. Using a `capacity` to new **vector** or **hash**, a least capacity elements without reallocating

5. SIMD

Most of the stuff mentioned are either micro optimization or even can be harmful(in certain cases).

1. Custom memory allocator does not impact perf in real world. You use it when you need extra features(security, debug/profiling capabilities and customizable behaviors etc).
2. Global variable has it's pros and cons and you should only use it when needed for reason other than micro optimization.
3. Inline small data is micro optimization and it has limitation where you must be able to predict the data is moving around the inline water mark. Otherwise you may end up with a strictly inferior data structure.
4. Reducing memory reallocation is mostly about reducing fragmentation and perf impact is on micro level. Over allocating upfront can possibly result in high memory usage so as point 3 whether you can predict the data size is the key.
5. In real world web app you better just choose the Rust crate that can do simd for you rather than optimize it yourself.

In general you should not take anything from tfb benchmark and simply consider it useful in real world. Context and use case determines how you optimize your code.

btw: xitca-web (bench code not including dependencies) does not do 2,3,5 and still remains competitive in micro bench can be used as a reference.

[franz1981]

@nbrady-techempower thanks, I've sent an email with the changes

[mindplay-dk]

No Bun or Deno in this one? I think these might be the two most wanted benchmarks this year.

[pavelmash]

There are both Bun and Deno here - but they only implement plaintext and json

[NateBrady23]

I'll be out of the office for a couple of days this week, so hoping to get the blog post and results up on the site by the end of the week or early next week at the latest; so last call for any other folks looking to get a blurb in the blog post.

[synopse]

I have written a blog post about TFB and object pascal - yes, we added our object pascal framework in round 22!
https://blog.synopse.info/?post/2023/10/31/Pascal-in-the-race%3A-TFB-Challenge-Benchmarks

And in fact, next week I will go to Germany for two conferences at EKON 27, speaking about TFB and comparing good old object pascal solutions to the most trendy using Rust, Go or JS. https://entwickler-konferenz.de/en/

Here is a paragraph about what we did on our mORMot framework in respect to TFB, through a lot of iterations:

About how we maximize our results for TFB, we tried several ways of accessing the DB (ORM, blocking, async), reduced the syscalls as much as possible, minimized multi-thread locks especially during memory access (a /plaintext request requires no memory allocation), and made a lot of profiling and micro-optimizations. The benefit of the object pascal language is obvious: it is at the same time a high-level language (with interfaces, classes and safe ARC/COW strings), safe and readable, but also a system-level language (with raw pointers and direct memory buffers access). So the user can write its business code with high level of abstraction and safety, but the framework core could be tuned to the assembly level, to leverage the hardware it runs on. Finally, OpenSource helped a lot having realistic feedback from production, even if the project and the associated FreePascal compiler are maintained by a small number of developers, and object pascal as a language is often underestimated.

[mindplay-dk]

There are both Bun and Deno here - but they only implement plaintext and json

ah, I see, there's a separate TypeScript folder.

how come? I mean, TS doesn't have a runtime - so in terms of performance, it's going to be dependent on whatever the emitted JS is running on. Any framework that supports TS also supports JS, and most of the frameworks in the JavaScript folder also support TS, so it doesn't seem very meaningful to break TS into a separate category from JS, I think? it's JavaScript, with types.

[Robbson]

The JavaScript frameworks I checked are still using the old NodeJS version 18.
Now that NodeJS 20 has LTS status, this should be used as the new default, especially because version 20 got really great performance improvements.

[NateBrady23]

@mindplay-dk

it doesn't seem very meaningful to break TS into a separate category from JS, I think? it's JavaScript, with types.

The separation isn't meant to mean anything other than the programming language the code is written in; not what it compiles/transpiles to.

@Robbson you are welcome to upgrade any languages and frameworks that you think are out of date. Even better if you can ping the maintainers while you do it.

[NateBrady23]

Round 22 has been posted. Blog post to follow early next week. Thanks everyone for your contributions!

[bhauer]

Thanks @nbrady-techempower for posting Round 22! Looking forward to the blog post.

[volyrique]

@nbrady-techempower Could you also add releases/tags for round 21 and 22 to this repository, please?

[hggq]

How to participate in this testing competition
https://github.com/hggq/paozhu
C++ Web framework

[NateBrady23]

@volyrique Ah, thanks for calling that out.

@franz1981 I think the email I thought was from you was from someone else. I'm about to get this blog posted. Would you mind resending to [email protected]

[franz1981]

@nbrady-techempower I'm [email protected] :) it should be (I'm Francesco, as first name)

[NateBrady23]

@franz1981 Ok, got it for real this time. Thanks!

[hiqsociety]

@nbrady-techempower possible to update the website on the latest pull request? i've submitted mine but was not updated.

golang's sprapp

[NateBrady23]

@hiqsociety The web results are for round 22. To watch continuous runs happen, that's on https://tfb-status.techempower.com

[volyrique]

@NateBrady23 Perhaps it is time to remove the message Round 22 official run has started... from the results dashboard 😉?

[libreom]

Hi, when will the blog post be published?

[firebank]

Hi, what is the precise version information of MySQL，MongoDB & Postgres？I can not find on the site or github.

[volyrique]

@firebank How precise would you like that information to be? You could look at the database Dockerfiles for a fairly accurate idea:

• MongoDB
• MySQL
• PostgreSQL

[ehsanziyaee]

Where is Drogon framework ? you should apply framework with best performance in last rounds, in new benchmark, too.

[volyrique]

Drogon didn't go anywhere, but it failed to build during the benchmark run for round 22.

[Anonymes1]

H

[Anonymes1]

``

[GigaDevelopper]

When you add Drogon Framework

[abgros]

@NateBrady23 I followed a link to the blog post and ended up here. Was the blog post ever published?

Edit: I found it! https://www.techempower.com/blog/2023/11/15/framework-benchmarks-round-22/