I2C bus speed of 400 kHz with ESP32-S3 produces "frame drops"

[r4y-h1ll]

Adafruit board: ESP32-S3 2MB PSRAM

Arduino IDE version: 2.0.1 and 1.8.16

Steps to reproduce the problem:
This is a copy of my problem description from the Adafruit forum https://forums.adafruit.com/viewtopic.php?t=196006, where it was suggested to open an issue here. Please understand, that I did not want to rephrase my whole issue description.

I got an Adafruit Quadrupler (2x2) to kind of organize all my Feathers/FeatherWings. I also bought one of the green CharlieWings. To test it I plugged all in and uploaded the swirldemo.ino (from the Adafruit_IS31FL3731 library) to the ESP32S3 Feather running at 240 MHz using Arduino IDE 2.0.1 as well as 1.8.16. However, the swirl was not running smooth, stuttering in irregular intervals. Running the same code on the Feather ESP32 V2 or HUZZAH32 did not show any stuttering and was super smooth, as you would expect.
I did a timing measurement and found it sometimes hanging for about 300 ms when calling 
drawPixel(x, y, intensity);.
I tried changing compilation parameters, but the only thing that had an effect was changing the CPU speed: 160 MHz made the stuttering even worse (more stuttering), 80 MHz or lower solved it (no stuttering at all).
Since it was only stuttering when drawPixel(...) was called my guess was it having something to do with the I2C configuration, so I took a look at the Adafruit_IS31FL3731.cpp. When calling begin(); on the object _i2c_dev->setSpeed(400000); is called. So this was my first thing to go for.
Changing the speed to 200000 -> uploading to the S3 @240 MHz -> swirling a bit slower, but no stuttering at all.
Changing the speed to 800000 -> uploading to the S3 -> swirling way faster, but again no stuttering.
Changing the speed to 300000 -> uploading to the S3 -> again no stuttering.
Changing the speed back to 400000 -> uploading to the S3 -> stuttering!!

For now I fixed it by adjusting the mentioned library from
_i2c_dev->setSpeed(400000);
to
#if defined ARDUINO_ADAFRUIT_FEATHER_ESP32S3
_i2c_dev->setSpeed(300000);
#else
_i2c_dev->setSpeed(400000);
#endif

I tested this with the absolute minimum of hardware and code to exclude side effects with other hardware/connections.
I also tested this with the OLED FeatherWing @400 kHz bus speed, which worked smooth as butter.