How compare result from different rounds?

[lestofante]

I am trying to estimate how those framework evolved in time, but if i compare the Round 18 with Round 20 composite scores, it looks like ALL framework did significantly lost performance.
Is this because they actually became slower, or is it because the Rounds score are not directly comparable?

[joanhey]

During the history of this benchmark, they were used different hardware.
Check it https://www.techempower.com/benchmarks/#section=environment

From Round 16 til 20 use the same hardware:

Three homogeneous Dell R440 servers each equipped with an Intel Xeon Gold 5120 CPU, 32 GB of memory, and an enterprise SSD. Dedicated Cisco 10-gigabit Ethernet switch.

But in Round 19 was included the spectre/meldown patch, for the security bug in CPUs.
That had a big impact on speed in all frameworks.

One example is Nginx web server alone:

Round	Plaintext
18	3,752,793
19	2,244,289

[lestofante]

thanks, i guess i will have to run the test locally if i want to have a normalized result between different version.

One example is Nginx web server alone

unless testing the same version could be some regression/new design that have different tradeoff, and that is exactly what i would like to see.
The framework i am interested in had some big changed between round 18 and 20 and i wanted to asses their tradeoff on performance level

[joanhey]

This necessary patch can result in between 5-30% performance overhead, in some CPUs. And the Xeon Gold 5120 is heavily affected.

More info:
https://www.phoronix.com/scan.php?page=article&item=linux-419-mitigations&num=4
https://www.phoronix.com/scan.php?page=article&item=spectre-meltdown-2&num=10