Why say webrtc: "WebRTC doesn't support H264 streams with B-frames"?

[zsfscode]

Question

I tried restreaming a H265 RTSP via a Jetson Nano and gstreamer.

When I use omx264enc is working fine.
When I use nvv4l2h264enc the webrtc server drop the stream with "WebRTC doesn't support H264 streams with B-frames".

I checked the both RTSP stream with ffprobe -show_frames, the stream dosn't have B frames only has I and P.

Where/Why sees mediamtx the "B-frames", how can I check that?

[aler9]

Hello, provide a server log and also a network dump of the traffic exchanged between GStreamer and the server, or alternatively a file that contains a sample stream generated with nvv4l2h264enc. You can generate a network dump in this way:

If the bug arises when using MediaMTX with an external hardware or software, the most helpful content you can provide is a dump of the data exchanged between the server and the target (network dump), that can be generated in this way:

1. Download wireshark (https://www.wireshark.org/)
2. Start capturing on the interface used for exchanging packets (if the server and the external hardware or software are both installed on your pc, the interface is probably "loopback", otherwise it's the one of your network card)
3. Start the server and replicate the issue
4. Stop capturing, save the result in .pcap format
5. Attach

[zsfscode]

Hello, attached the mediamtx debug log and pcapng record between the gstreamer, mediamtx and client.
logandpcapng.zip

[aler9]

@zsfscode both dumps you provided seem to be corrupted. Regarding the GStreamer pipeline with nvv4l2h264enc, my advice is to set the profile to baseline instead of manually controlling B-frames, this can be achieved by using the profile flag:

nvv4l2h264enc profile=0

[RobotnickIsrael]

WebRTC by design works on H264 baseline and main profiles, messing streams with Iframes and PFrames only. This is because webrtc is designed for low latency, and bframes forces latency in order to wait for a future frame.

Transcode ur file with ffmpeg or gstreamer and remove the bframes.
You can use ffprobe to write you the type of frames in the file

[neilyoung]

I have some doubts regarding the statement "WebRTC doesn't support H264 streams with B-frames" This statement is correct, had to learn it. Correct in the way, that none of the todays browsers like it. I'm also seeing these messages (and the harsh reaction of mediamtx on this) too from time to time. Also in my case nvv4l2h264enc is involved as source of the stream (via RTSP), even though the encoder is advised to not send B-Frames.

I admit that I never had any problems with B-Frames in the past, even though I have a long WebRTC experience. I also don't believe, that WebRTC generally has a problem with B-Frames at all, otherwise I should have seen that in other scenarios, maybe. So I checked the part, which is responsible for the abort and changed it in internal/servers/webrtc/session.go:

				if !firstReceived {
					firstReceived = true
				} else if tunit.PTS < lastPTS {
					//return fmt.Errorf("WebRTC doesn't support H264 streams with B-frames")
					fmt.Println("WebRTC doesn't support H264 streams with B-frames")
				}
				lastPTS = tunit.PTS

I'm still seeing the message very rarely, but the WebRTC connection now no longer terminates.

I'm also asking myself, if this problem couldn't just be caused by an mis-order in the RTP receive stream...

At least I cannot see any negative impact when I'm just ignoring that...

[neilyoung]

Created a PR #3196

[aler9]

I'm also asking myself, if this problem couldn't just be caused by an mis-order in the RTP receive stream...

Regarding this, there's a component that performs packet reordering and removal of duplicates (https://github.com/bluenviron/gortsplib/blob/main/pkg/rtpreorderer/reorderer.go) so it is probably not the issue.

I admit that I never had any problems with B-Frames in the past, even though I have a long WebRTC experience. I also don't believe, that WebRTC generally has a problem with B-Frames at all

It's not WebRTC by itself that has problems with B-frames, it's the implementation inside most browsers. Streams get frozen as soon as the timestamp is not increasing anymore. Try removing the error message and you'll see for yourself.

[neilyoung]

It's not WebRTC by itself that has problems with B-frames, it's the implementation inside most browsers. Streams get frozen as soon as the timestamp is not increasing anymore. Try removing the error message and you'll see for yourself.

Yes, I did that. I removed the "fatal" and made it a warning. I'm forwarding the packets to a Chrome instance, which receives via WHEP. There is no issue..

[TimoKettunen]

When there are frequent B-frames, the fix above freezes the video time to time.
returning nil in the B-frame case removes the freezes but impacts on video quality. More artifacts are generated and slight jerking starts to appear.

				if !firstReceived {
					firstReceived = true
				} else if tunit.PTS < lastPTS {
					fmt.Println("WebRTC doesn't support H264 streams with B-frames")
					return nil
				}

[neilyoung]

I was just trying to get rid of the harsh reaction of tearing down everything just because of such a frame. I could live with artefacts, in fact, RTP losses look similar, so where is the point of tearing down the entire connection? Education? :)

[TimoKettunen]

You can get rid of b-frames altogether by transcoding but that adds a bit latency and incorporates ffmpeg. In mediamtx.yml:

paths:
  stream0:
    source: publisher
    runOnInit: ffmpeg  -r 30 -i rtsp://@<sourec-addr>/stream0 -c:v libx264 -bf 0 -f rtsp -rtsp_transport udp -preset superfast rtsp://localhost:8554/stream0

[neilyoung]

@TimoKettunen Thanks, one question regarding this, because it looks like "re-encoding" or at least "slight re-encoding". Any impact to the CPU of the host system and to overall latency known compared to the plain forwarding?

[TimoKettunen]

Of course re-coding causes latency and cpu load, depending on host HW. I do not think that mediaMTX should do that since it can be done with separate tools like above with ffmpeg. I would just return nil and instruct users to use ffmpeg.
Performance with ffmpeg can be improved if you happen to have nvidia GPU:

runOnInit: ffmpeg  -r 30 -c:v h264_cuvid -i rtsp://@<source>/stream0 -c:v h264_nvenc -bf 0 -f rtsp -rtsp_transport udp  rtsp://localhost:8554/stream0