Add WebRTC (WHIP) output support

[kc5nra]

Co-authored-by: Sean DuBois [email protected]
Co-authored-by: DDRBoxman [email protected]
Co-authored-by: tt2468 [email protected]

First and foremost, thanks to the original author of the draft PR for webrtc Colin Edwards (DDRBoxman).

Description

This PR adds WebRTC support to OBS. WebRTC is an alternative to RTMP that is being adopted by more services. Adding WebRTC support to OBS would benefit OBS and its users greatly. These are some of the highlights. Following this I also have a FAQ for questions we have already received regarding this PR.

This initial version is intentionally simplified to only support output. Later PRs will add source support.

Specifically, this adds a plugin that extends support for:

• WebRTC Service type with recommended settings for maximum compatibility with webrtc peers
• WebRTC Output

Motivation and Context

Subsecond Latency

With WebRTC we are able to achieve sub second latency. With our initial measurements we see ~120 Milliseconds from Broadcaster to Playback. We believe we can continue to bring down this number as well.

Interactive latency lets broadcasters create new experiences they couldn’t before. OBS users can now create interactive experiences with their viewers. This would allow talk shows and other productions that require conversational latency to use OBS.

Remove FTL from OBS

FTL is a protocol that was created for Mixer. It provided sub-second latency and was used by a few Open Source projects and streaming services. The protocol has been abandoned and it would be nice to remove it from OBS. Adding WebRTC would make that possible since it provides the latency required in addition to the many benefits inherent in the WebRTC stack like encryption, network topology strategies, robust congestion control, etc.

Latest in Video Compression

WebRTC will let OBS users experiment with newer codecs. WebRTC is flexible in what codecs it supports so users could use H265 and AV1.

AV1 users are reporting a 50% reduction in bandwidth required.

Broadcast all angles

WebRTC supports multiple video streams in the same session. OBS could experiment with broadcasting multiple scenes at once! Viewers could then switch between scenes or different views in the same game.

Simulcast

WebRTC allows broadcasters to upload multiple streams of different quality. OBS users could upload ‘high’, ‘med’, and ‘low’ streams themselves. This would reduce the server costs greatly, allowing more

Peer-to-Peer (if you need it)

With WebRTC OBS users could share their video without needing a video server at all! WebRTC can establish a P2P connection so users could send video from OBS directly to their users. No more worrying about setting up special servers to ingest and transcode.

In the future we could add a WebRTC source. Then broadcasters could add WebRTC sources to their scenes. This means low latency collaboration with no backend required!

How Has This Been Tested?

This has currently been tested on Linux and Mac with locally run broadcast-box as well as remotely hosted WHIP endpoints.

Configurations tested:

• Mac OSX M1 Monterey
• Debian Sid

Original draft PR: #7192
Closes #7192

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[kc5nra]

FAQ

How can I use this today?

Multiple companies provide WebRTC Broadcast/Playback today. Cloudflare and Millicast are two popular options.

Gilmesh has created and is hosting a public WebRTC server at https://b.siobud.com/. You can run this locally for developing and testing as well https://github.com/glimesh/broadcast-box

Why libdatachannel as the WebRTC library?

WebRTC has many libraries that we could have used. The first one we evaluated was Google's implementation known as libwebrtc.
The time required to fetch+download would have been too much of a burden for the OBS code base. These metrics come from a Macbook M1 with a 400 Mb/s connection.

Fetching the source of libwebrtc: 11 minutes 20 seconds
Installing the dependencies: 3 minutes and 30 seconds
Building libwebrtc: 7 minutes
Size added by libwebrtc: 35 megabytes

This caused us to evaluate other options. We arrived at https://github.com/paullouisageneau/libdatachannel and it had everything we needed! It is a C++ WebRTC implementation and uses CMake. It adds very litle impact to the project.

Additional Build time: 9 seconds
Size Added: 11 Megabytes

What is WHIP?

WHIP is the process that WebRTC uses to establish a session between two WebRTC Agents (Client/Server in our case). The actual definition of it can be found at https://www.ietf.org/archive/id/draft-ietf-wish-whip-05.txt

[DDRBoxman]

[RytoEX]

For some reason, build-aux/modules/50-libdatachannel.json is using libdatachannel paullouisageneau/libdatachannel@f66f281 while obs-deps is using paullouisageneau/libdatachannel@506961b. Is that intentional?

[pkviet]

For some reason, build-aux/modules/50-libdatachannel.json is using libdatachannel paullouisageneau/libdatachannel@f66f281 while obs-deps is using paullouisageneau/libdatachannel@506961b. Is that intentional?

That's an oversight
That being said, this is not very consequential. I plan on working on a full msvc compile using Pat's scripts which will allow syncing to last libdatachannel commit on all 3 platforms.

[ma3uk]

It works well, but periodically the image is showered with artifacts (tested with NVENC, QSV and x264). Great job, I'm waiting for support for B-Frames and other codecs

[norihiro]

After the commit dd39218, obs gets crashed when opening the settings dialog. Steps to reproduce is as below.
OS: Linux

1. Build with -DENABLE_WEBRTC=OFF.
2. Start obs. At the first-time wizard, configure with 'Optimize just for recording, I will not be streaming'. Exit obs.
3. Start obs and open settings.

Log shows a line below and obs crashed immediately.

error: Service 'whip_custom' not found

Backtrace is as below.

#0  obs_service_get_supported_video_codecs (service=0x0) at /home/kamae/repo/obs-studio/libobs/obs-service.c:465
#1  0x00000000007f2a91 in OBSBasicSettings::ResetEncoders(bool) (this=0x7ffe97518168, streamOnly=false)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings-stream.cpp:1552
#2  0x000000000072087a in OBSBasicSettings::LoadOutputSettings() (this=0x0, this@entry=0x7ffe97518168)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings.cpp:2477
#3  0x00000000006f8518 in OBSBasicSettings::LoadSettings(bool) (this=0x7ffe97518168, changedOnly=false)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings.cpp:3315

[Sean-Der]

Sorry about that @norihiro i will have that fixed today!

[RytoEX]

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.

paullouisageneau/libdatachannel@v0.19.0-alpha.1...v0.19.0-alpha.3

[pkviet]

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.

paullouisageneau/libdatachannel@v0.19.0-alpha.1...v0.19.0-alpha.3

Yeah, i was just suggesting that i'd make a new change to the commit which is checked out, before the first beta, in order to remove the current libdatachannel patch so maybe it's not necessary to multiply the PRs ? Your call

[RytoEX]

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.
paullouisageneau/[email protected]

Yeah, i was just suggesting that i'd make a new change to the commit which is checked out, before the first beta, in order to remove the current libdatachannel patch so maybe it's not necessary to multiply the PRs ? Your call

Personally, I'd prefer the deps were aligned sooner rather than later. Less time for people to run into obscure issues. Want CI builds to be in the best shape we can manage, and a one-line PR is simple enough to review.

[fairbairn]

Would you mind sharing your OBS settings used to achieve this 120ms glass to glass latency? We've set up OBS to aggressively do the same, similar setup (we think), but cannot get under 200ms (localhost, with broadcast-box). We'd like to replicate your setup for our experiments.

[Sean-Der]

@fairbairn What encoder (and settings) are you using?

I saw this latency when I did a tune of zerolatency with x264

[fairbairn]

x264, main, 5000K, CBR, 1080p, 30fps, ultrafast, zerolatency

did you use main or baseline?

also, were you streaming audio as well, or just video?

[fairbairn]

@Sean-Der

We have been unable to replicate this latency, even locally while transmitting through broadcast-box to remove all network influences.

We're unable to get below 210-260ms in our various tests, and we always tune for low latency on the video encoder by default (x264 as well as NVIDIA).

Can we get the precise setup you used (audio and video) from your test where you achieved 120ms so that we may reproduce in our own lab?

In our tests, we're accompanying the video with audio, which may have an influence, but we'd very much like to reproduce your baseline before we tune from there.

Thank you.

[Sean-Der]

Hi @SetoKaiba I have a build of Simulcast available here

I am actively working on it so will have bugs, thank you!

[Sean-Der]

@fairbairn

Sorry I missed this! It might be better if you reach out on OBS discord. I get a lot of github notifications, easy to miss things :/

• M1 Mac
• x264 enc
• tune=zerolatency
• Run Broadcast Box on localhost