metric</th>	main (baseline)</th>	RACK</th>	delta</th></tr></thead>
goodput</td>	234.55 Mbps</td>	316.42 Mbps</td>	+34.9%</strong></td></tr>
CPU time (`cpu_seconds</code>)</td>`	0.0560 s</td>	0.0441 s</td>	−21.3%</strong></td></tr>
goodput / CPU-second</td>	4,189</td>	7,177</td>	+71.3%</strong></td></tr>
latency p50</td>	16.37 ms</td>	11.86 ms</td>	−27.5%</strong></td></tr>
latency p99</td>	36.95 ms</td>	27.84 ms</td>	−24.6%</strong></td></tr> </tbody></table> That +71% throughput-per-CPU</strong> is simply the goodput measured (Mbps) divided by the run's `cpu_seconds</code>. Non-rack cruised at ~234 Mbps using ~0.056 CPU seconds (~4,189 Mbps/CPU-s), while RACK sustained 316 Mbps with ~0.044 CPU seconds (~7,177 Mbps/CPU-s). That gap is the proof that rack delivers ~71% more work per unit of CPU.</p>` Test setup</a></h3> To test RACK, we ran these test profiles to compare how RACK performs against main (baseline):</p> max-burst</strong> - "how fast can we go" with no delay, loss, or reordering; it targets the raw transport path:</li> </ul> Goodput jumps +34.9% (234 ->316 Mbps) while CPU seconds drop by 21% (0.056 ->0.044 s) and p50/p99 both fall by ~25%. RACK now delivers ~71% more Mbps per CPU-second.</p> handshake</strong> - same burst pattern, this time including</strong> the COOKIE/SHUTDOWN handshake, so we exercise setup timers:</li> </ul> Goodput climbs +15% (237 ->272 Mbps) while latency stays basically flat (15.65 ->15.99 ms for p50, 35.27 ->33.25 ms for p99), which confirms that the faster throughput comes without slower ACK paths.</p> unordered-late-low-rtt</strong> - minor delay/jitter (10 ms) but unordered delivery to simulate packet trains with mild disorder:</li> </ul> Minor latency and throughput noise. Both branches still pass but RACK keeps the delivery steady despite small unordered bursts.</p> unordered-late-high-rtt</strong> - large RTT/jitter (180 ms / 60 ms) with unordered delivery, so we can watch how the stack copes with latency spikes:</li> </ul> Very high latency due to the profile, but RACK keeps throughput comparable while completely avoiding any regressions.</p> unordered-late-dynamic-rtt</strong> - fluctuating RTT (40 ms base ±180 ms jitter) with unordered delivery to mimic burst-y network dynamics:</li> </ul> Both branches pass with no noticeable regressions from RACK, which shows that each branch handles jitter swings fine.</p> congestion</strong> - ordered delivery with 2% loss and modest delay/jitter to stress-tests congestion control and SACK-driven recovery:</li> </ul> The loss-handling path stays green and RACK doesn't use extra CPU compared to master which shows the +35% clean-case gain doesn't cost the loss profile.</p> retransmission</strong> - ordered with 5% loss and 20 ms jitter to force fast-retransmit/TLP scenario:</li> </ul> The fault case still hits retries and RACK's CPU profile actually shows more JSON/packet-logging work but that's what we expect during retransmit storms.</p> reorder-low</strong> - unordered with 1.5% loss plus deliberate reordering to exercise scheduler/queue behavior under loss＋reorder:</li> </ul> Goodput improves +44% (1.79 Mbps -> 2.58 Mbps) and the run finishes faster (~3.70 s vs 5.34 s), so RACK dominates the low-rate reordering scenario.</p> burst-loss</strong> - unordered with 4% loss and 50 ms jitter to push retransmit/recovery under heavy loss bursts.</li> fragmentation</strong> - oversized payloads require chunk fragmentation/reassembly to verify large-message handling:</li> </ul> Nothing improved or got worse.</p> media-hevc</strong> - A real-world use case with video: one-way stream, paced HEVC frames (~25 fps), 3% loss, ~1200-byte max payload, across a ~13-14 Mbps link (taken from a real-world use case of sending DRM media over WebRTC datachannels) to ensure sustained media delivery works.</li> </ul> RACK hits 12.90 Mbps goodput in 2.14 s (100% delivery) while the main branch streaming to the RACK branch sits at 11.34 Mbps in 4.66 s. That's a 2x faster finish!</p> We also have 3 negative tests to ensure that any corruptions or errors are still being caught:</p> fault-checksum</strong> - corrupts every 7th DATA chunk's checksum so receivers must drop it and log the error.</li> fault-bad-chunk-len</strong> - mangles the chunk length field every 7th chunk to validate length checks/parsing.</li> fault-nonzero-padding</strong> - corrupts padding bytes every 7th chunk so padding validation and chunk isolation logic are exercised.</li> </ul> In both branches, these cases fail (as desired), which confirms that both branches detect the corruption and that there is no regression in behavior.</p> CPU flamegraphs</a></h3> The flamegraphs below show the CPU profiles for max-burst runs. You can see in the metadata that the RACK profile captured 20ms of samples</strong> (9.95% of 201.08ms duration) versus the master profile's 40ms of samples</strong> (19.86% of 201.45ms duration) – exactly half the sample count for a similar duration, which directly supports the efficiency claims.</p> </p> </p> Why RACK behaves better (not just "goes faster")</a></h3> RACK changes how</em> SCTP decides that something is lost and when it sends probes, so it wastes less work fixing problems that never really happened:</p> Instead of keying almost everything off "three missing reports or an RTO fired"</strong>, RACK uses time-based loss detection</strong>: it looks at when chunks were last SACKed/ACKed and infers loss from elapsed time and the pattern of tail acknowledgments.</li> Tail Loss Probes (TLP)</strong> send a cheap "sample" chunk at the end of a burst to flush out late ACKs. If the receiver really did get the data, it answers and the sender avoids a full retransmission storm, otherwise, the probe doubles as the retransmission you needed anyway.</li> </ul> In practical terms that's what the profiles and metrics are showing:</p> We still see the same hot stack (vnet.(*chunkUDP).UserData</code>, runtime.memmove</code>, a thin layer of runtime/type helpers) in both master and RACK. It's the normal packet I/O path.</li> With RACK, that stack is exercised fewer times per unit of useful data</strong> because there are fewer spurious retransmits and fewer "just in case" timer expirations.</li> That's exactly how we get more goodput, lower latency, and smaller CPU profiles at the same time</strong>: RACK spends less CPU "arguing with the network" and more CPU pushing real user data through SCTP.</li> </ul> Spec-aligned ACK behavior and testing</a></h3> Using SCP testing tool we were able to find some issues includes:</p> In the initial RACK implementation, handling of transitions from high to low RTT was suboptimal due to the implementation using a global minimum for recent RTT measurements instead of a windowed minimum (the latter approach is only a "SHOULD" in RFC 8985 section 6.2.1</a>). Atsushi Watanabe quickly identified it and we resolved the issue.</p> </li> The earlier version of RACK implementation also handled packet reordering poorly and consumed more CPU than non-RACK. This was corrected by implementing improved active RTT measurement, following the approach described in Weinrank's work, see p. 120</a>.</p> </li> A minor bug was discovered (and fixed) in the initial RACK implementation where the latest RTT was not measured for every packet.</p> </li> We also found that Pion SCTP did not send a SACK immediately after a TSN gap, causing RACK to perform worse under moderate reordering. After fixing this behavior to align with RFC 4960 section 6.7 (surprisingly only a "SHOULD")</a>, reordering test cases showed a ~30% improvement.</p> </li> </ul> Looking forward</a></h2> Keep an eye out for even more improvements and benchmarks from our improved SCTP implementation using real-world data, as well as how we're doing it in an upcoming blog post!</p> Thanks for reading!</p> Credits</a></h2> Huge thanks to the following for making this possible:</p> Jo Turk</a> for introducing me to Pion, making SCP, answering countless questions, and so much more.</li> Sean DuBois</a> for making Pion, finding Felix Weinrank's thesis</a>, and endless encouragement.</li> Srayan Jana</a> for helping to bounce around many ideas.</li> Atsushi Watanabe</a> for reviewing and catching the global minimum vs windowed minimum issue in the RACK PR.</li> And many more people along the way!</li> </ul> WebRTC Survives When You Walk Out Kevin Wang — Fri, 17 Oct 2025 00:00:00 +0000 Networking for a video call sounds simple. You send your video to the server and server sends video back. But that mental model breaks down in the real world. A connection that starts on stable home Wi-Fi may suddenly need to continue over cellular as someone walks out the door. The real challenge isn’t just connecting two points! It’s staying connected as the network beneath you changes.</p> With automatic renomination</a>, Pion will now handle that completely seamlessly, allowing your media to flow without skipping a beat. It works by periodically detecting when new connection candidates are available and automatically switching to the highest quality candidate, taking advantage of a feature in WebRTC called renomination</em>.</p> Let's dive into how that works.</p> How does WebRTC establish connections?</a></h2> Connections between two peers in WebRTC are established by exchanging candidates</strong> between the two peers in a signaling phase. These candidates can be host, server reflexive, peer reflexive, or relay candidates. The details of the various types of candidates are better explained in WebRTC for the Curious</a>, but the relevant detail is that for each candidate, both peers open up ports to communicate.</p> </p> Eventually, the controlling</strong> peer picks a candidate and tells the controlled</strong> peer to use it in a Binding request. This candidate becomes the selected candidate that media flows over. After this point, the unused ports are closed.</p> </p> But wait! This handshake is pretty slow. To speed things up and get media flowing faster, WebRTC allows media to flow over any</em> candidate before one is formally nominated. This shaves off a few round trips between the peers, reducing connection latency, in a process called aggressive nomination</a>.</p> Changing candidates with renomination</a></h2> Now, what if we never</em> commit a candidate? The ability to send media over any valid candidate is curious, and in fact quite useful because it means we have multiple paths to take advantage of. This is called renomination</strong>.</p> Under renomination, the controlling peer tells the controlled peer that it wants to be able to renominate candidates. When the controlled peer receives a binding request, instead of closing the ports, it leaves them open. This allows the controlling peer to elect a new candidate at will, which the controlled peer will accept media over, effectively enabling us to switch candidates mid-stream.</p> </p> That can sound complex to negotiate but it amounts to a flag in the connection handshakes that ensures that both parties are aware of the desire to renominate, meaning the implementation of this feature in Pion amounts to keeping track of a counter</a>.</p> Now Pion is taking advantage of WebRTC's aggressive nomination state, keeping candidates open, and has the ability to pick a new candidate.</p> Making it automatic</a></h2> Ideally, the WebRTC connection can do this automatically so the user doesn't have to pick. After all, manually renominating candidates seems useful but is a tedious user experience. Let's do it automatically!</p> To make this work, we need to solve two problems: knowing when the current candidate gets worse, and discovering when completely new candidates appear.</p> Detecting when candidates get worse:</strong> Turns out, this is already built into WebRTC. During the first connection establishment phase, candidates are pinged, evaluated, and ranked based on round trip time, packet loss, and type of candidate. This same ranking can be performed periodically instead of only at initialization. So, the connection can run this on a loop and automatically renominate candidates when the state of the network changes.</p> Discovering new candidates:</strong> What happens when a completely new candidate comes online? For example, if I connect to a wifi network mid-call, the phone receives a completely new IP address.</p> This, too, is built into WebRTC, in the form of trickle ICE! If trickle ICE is enabled, candidate exchange can continue after the initial connection handshake until the two sides declare that they've exhausted the candidate search. Under automatic renomination, we go one step further and disable the exhaustion declaration</strong>, periodically checking for new candidates instead. When a new candidate arrives, we use the normal trickle ICE mechanism to exchange it between the two peers.</p> Putting it all together</a></h2> And that ends up being enough! In summary, automatic renomination makes the following tweaks:</p> Signal renomination between the peers</li> Keep all candidate ports open instead of closing them</li> Disable trickle ICE termination</li> Periodically ping all candidates and rerank them</li> Renominate when a better candidate arises</li> </ul> With these pieces in place, Pion can seamlessly transition between different network links. If you want to see it in action, check out this example</a>!</p> One really nice thing that I personally liked about this feature when implementing it is that it felt like automatic renomination is a clever reading of the specification: it doesn't need to reinvent a whole new mechanism and RFC.</p> In fact, one can go one step further and send media over all</em> candidates, because under aggressive nomination they are all valid. This, actually, is connection bonding for free! There are some practical considerations to getting this working well though, but that's another blog post.</p> Making a game with Pion Srayan Jana — Tue, 09 Sep 2025 00:00:00 +0000 (The following was adapted from a talk I gave at DWeb Weekend 2025</a> at the Internet Archive in San Francisco on August 17, 2025)</p> Simplifying WebRTC Datachannels for Games</a></h1> First of all, to get some stuff out of the way, instead of using the "super complicated" WebRTC datachannels, why don't we use something simpler?</p> Why Not Use Websockets?</a></h2> Too slow</strong>: Most games use UDP with a reliability layer on top.</li> Suitable for turn-based games</strong>: Fine for games that are turn-based/slow-paced, like Runescape.</li> For more details, see: UDP vs TCP</a></li> Client-Server Connection</a></li> </ul> </li> </ul> The Case for Web Transport</a></h2> Replacement for Websockets, uses QUIC instead of TCP.</li> However, it has been in development for a long time, and I want to get started on making games now.</li> Current issues: Caniuse WebTransport</a></li> Official demos on GitHub</a> do not work on Firefox on Windows.</li> GitHub Issue</a></li> Firefox Bugzilla</a></li> </ul> </li> Potential</strong>: Will probably solve most problems once finished and would be the best choice for making a multiplayer game on the web.</li> </ul> Datachannels: A Hidden Gem of WebRTC</a></h2> Feature</strong>: Lets us send unreliable packets over the web using SCTP.</li> Advantage: We don’t need WebTransport; we can use this today.</li> They were created in part for exactly this use case See the reasoning for unreliable datachannels in the official specification itself</a></li> Not to mention that one of the original implementers of WebRTC wanted to enable this usecase</a> </li> </ul> </li> </ul> Benefits and Drawbacks of WebRTC</a></h2> Benefits</a></h3> Host flexibility</strong>: Do not need to host servers - players can make their own. No more need for port forwarding or Hamachi!</li> </ul> </li> Minimal server requirements</strong>: Only need one server for signaling.</li> Community-supported specification</strong>: There are many options for making your own WebRTC-based app.</li> </ul> Drawbacks</a></h3> Complex setup</strong>: Setting up WebRTC is challenging.</li> Server dependence</strong>: You need to host or use two different servers (Signaling + STUN/TURN). Can use Google’s STUN, but combining signaling and STUN into one server would be nice.</li> </ul> </li> </ul> WebRTC Implementations</a></h2> libwebrtc</a> - (a fork of) the original WebRTC implementation.</li> libdatachannel</a> - C/C++.</li> Pion WebRTC</a> - Go.</li> webrtc-rs</a> - Rust (using Tokio Runtime).</li> str0m</a> - Rust (sans-io).</li> sipsorcery</a> - C#</li> GameNetworkingSockets</a> Uses ICE and STUN/TURN for Peer to Peer.</li> Created by Valve and made for Steam, which means if you've ever played a game like Counter Strike or Deadlock, you've already seen this library in action!</li> </ul> </li> </ul> Existing Game Networking Libraries Using WebRTC</a></h3> Geckos.io</a> A client-server abstraction for WebRTC Datachannels written in Node.js.</li> Past experience: Really nice but a bit inefficient. See GitHub Issue</a>.</li> </ul> </li> Netlib</a> - Peer-to-peer WebRTC datachannel library for TypeScript.</li> Matchbox</a> - WebRTC datachannel library for Rust, compiles to both native and WASM.</li> PeerJS</a> - Great for browser-only apps/games, not specifically for game networking.</li> Netplayjs</a> - Untested but seems to work well.</li> Godot Engine's WebRTC Native</a> - Official Godot bindings to libdatachannel for Godot’s multiplayer API.</li> </ul> Real-World Uses</a></h2> WebXash3D</strong>: Reimplementation of Half-Life 1 + Counter-Strike 1.6 using WebRTC. GitHub WebXash3D</a></li> Xash3D Launcher</a></li> Counter Strike 1.6 on the web</a></li> </li> </ul> </li> Hypersomnia</strong> - Open source 2D shooter made with C++ and libdatachannel Has cross platform capabilities between both The native Steam port</a></li> And in the browser (On CrazyGames</a> and the official website</a>)</li> </ul> </li> Github Link</a></li> </ul> </li> </ul> Why Go for Development?</a></h2> Efficient: It just gets the job done.</li> Pion: Has a pure-Go implementation of WebRTC.</li> Minimal dependencies: No need for external dependencies like OpenSSL.</li> </ul> Why Not Rust?</a></h2> Preference: I like Rust, but Go has a more active scene/easier help for WebRTC.</li> Speed of development: It's faster to code in Go compared to Rust.</li> </ul> Why Ebitengine for Game Development?</a></h2> Website Link</a></li> Versatile</strong>: 2D game engine that works across various platforms. The creator, Hajime Hoshi</a>, has ported the engine (and the Go language!) to the Nintendo Switch</a> and other game consoles</li> </ul> </li> Battle Tested</strong>: Has been used for actual games like From Madness with Love</a> Steam Link</a></li> </ul> </li> Coral & The Abyss Steam Link</a></li> Nintendo eShop Link</a></li> </ul> </li> Rakuen Steam Link</a></li> Nintendo eShop Link</a></li> </ul> </li> And more on the Ebitengine Showcase Page</a></li> </ul> </li> Potential Crossplay</strong>: You could make a cross platform multiplayer game in pure Go that works between the Nintendo Switch, PC, and the browser!</li> </ul> Official Pion Example</a></h2> We now have an official example using Ebitengine in the Pion example-webrtc-applications</a> repository</li> This includes an bundled in signaling server so you can host your own lobby, and connect with another player.</li> Github Link to Game</strong></a></li> PR where it was merged in</strong>: Example WebRTC Applications</a></li> Limitations</strong> Right now, this can only support two players on the same computer Could probably work between two different computers, but we would need to figure out how to setup CORS properly for the signaling server</li> The machinery is there to support more than two players in a lobby, but as of writing this article it is currently hardcoded</a> to just two </li> </ul> </li> </ul> </li> </ul> Other Games That Use WebRTC</a></h2> Tough Love Arena</a></li> Counterpick Labs</a></li> 2D Soccer</a></li> </ul> Possibilities with WebRTC</a></h2> Hosting a Minecraft-style game without dedicated servers or port forwarding.</li> Players host everything themselves: no need for VPN/Hamachi.</li> </ul> Community Acknowledgements</a></h2> A big thanks to: The Pion Discord</a></li> The libdatachannel Discord</a></li> The Ebitengine Discord</a></li> The Rust gamedev Discord</a></li> </ul> </li> And many more who made this journey possible. I stand on the shoulders of giants.</li> </ul> FEC with Pion Jingyang Kang — Mon, 23 Jun 2025 00:00:00 +0000 We are excited to have FEC encoding support available in the pion/webrtc v4.1.2 + pion/interceptor v0.1.38. In this blog post, we'll explain what FEC is and how to implement it with Pion.</p> If you already know FEC, feel free to skip to FlexFEC in Pion</a>.</p> </blockquote> Quick background: What is FEC?</a></h2> Forward Error Correction (FEC) is a proactive loss-recovery mechanism:</p> The sender transmits extra mathematically derived parity packets (e.g. xor of the media packets) along with each block of media packets. If the receiver later discovers that one or even more packets are lost, it can use the parity packets to recover the lost packets, using the XOR result of the received packets and the parity (repair) packet, all without waiting for the lost packets to be retransmitted.</p> A simple example:</p> Data packets</th> Parity packet</th> Recovery logic</th></tr></thead> A</code> B</code> C</code></td> P = A ⊕ B ⊕ C</code></td> If B</code> is lost, receiver computes B = A ⊕ C ⊕ P</code></td></tr> </tbody></table> Because parity travels with the media, recovery happens locally and immediately. The trade‑off is more data on the link.</p> We Already Have NACK, Why FEC?</a></h2> Alice is sending a video stream to Bob with WebRTC. In an imaginary perfect network condition in which no packet loss happens and the network delay is 0 ms, all Alice needs to do is encode the video, pack RTP packets, and send them to Bob. What Bob needs to do is simply depacketize RTP packets and decode the payload into video frames. Everything is great except this imaginary perfect network condition doesn't exist.</p> sequenceDiagram</span></span> title "Perfect Network"</span></span> participant Alice</span></span> participant Bob</span></span> Alice->>Bob: Video Packet n</span></span> Alice->>Bob: Video Packet n+1</span></span> Alice->>Bob: Video Packet n+2</span></span> Note right of Bob: Great! :)<br/>But not reality</span></span></code></pre> Let's gradually introduce more factors into our network, to make it closer and closer to the real internet we live in. First, let's add packet loss while keep network delay still at 0 ms. With packet loss, not every packet Alice sends is guaranteed to be received by Bob. So, we need a mechanism to recover these lost packets. A straight forward approach is to make Alice resend the lost packets again. In order to notify Alice which packet is lost and needs to be retransmitted, Bob should explicitly send message to Alice, containing the sequence number of lost packets. This is NACK. With NACK in our lossy but no-delay network, all packets can be retransmitted immediately after Bob sends a NACK request. Problem solved? In this case, yes. But in the real world, no.</p> sequenceDiagram</span></span> title Using NACK</span></span> participant Alice</span></span> participant Bob</span></span> Alice->>Bob: Video Packet n</span></span> Alice->>Bob: Video Packet n+1</span></span> Alice->>Bob: Video Packet n+2</span></span> alt Packet n lost</span></span> Bob-->>Alice: NACK n</span></span> Note right of Bob: Extra RTT(s) here</span></span> Alice->>Bob: RTX Packet n</span></span> end</span></span></code></pre> Next, we are also adding delays to our imaginary network. At this point, it basically has all the characteristics of real world networks. Can our current mechanism tackle network delay + loss without hurting transportation quality? Sadly, no. If Alice packs a frame into 10 packets, Bob can't decode the frame even if only 1 packet is lost. And since NACK takes at least 1 additional RTT (more than 1 when retransmitted packets are lost too), this frame is delayed on Bob's side. To make things even worse, in modern video codecs, in order to increase compression ratio, encoders use a technology called forward prediction</strong>, which means subsequent frames relies on the data of previous frame to decode. For any frame delayed due to network loss, all subsequent dependent frames will be delayed. This can cause noticeable jank for Bob, which is bad.</p> sequenceDiagram</span></span> title Using FEC</span></span> participant Alice</span></span> participant Bob</span></span> Alice->>Bob: Video Packet n</span></span> Alice->>Bob: Video Packet n+1</span></span> Alice->>Bob: Video Packet n+2</span></span> Alice->>Bob: FEC Packet</span></span> alt Packet n lost</span></span> Bob->>Bob: Recover n from n+1, n+2, and the FEC Packet</span></span> Note right of Bob: No extra RTT(s) here</span></span> end</span></span></code></pre> FEC is a mechanism that can mitigate this. FEC stands for Forward Error Correction</strong>. Basically, it means that we can send more data to recover from potential packet losses. With FEC enabled, Alice does not only send 10 video packets to Bob, but also a few redundant packets, containing data XORed from video packets. When packets get lost during transmission, Bob can try to recover it with redundant data from FEC packets. In this case, no additional RTT is needed to recover packets lost, so video streaming is not affected.</p> Recovery groups</a></h2> On the sender side</a></h3> Media packets are grouped for protection, for example:</p> Group 1: M1, M3, M5 protected by F1</li> Group 2: M2, M4 protected by F2</li> </ul> </p> On the receiver side</a></h3> During the transmission, some media packets might be lost:</p> Received FEC packets are used to reconstruct missing data:</p> F1 recover any lost packet from Group 1</li> F2 recover any lost packet from Group 2</li> </ul> </p> FEC Mechanisms in WebRTC</a></h2> In WebRTC there are two main RTP based forward-error-correction (FEC) mechanisms. They both generate XOR parity packets, but the way they're signalled and routed is very different:</p> Feature</th> FlexFEC (RFC 8627</a>)</strong></th> ULPFEC + RED (RFC 5109</a>)</strong></th></tr></thead> Transport</strong></td> Travels on its own SSRC and payload-type, SDP advertises an extra repair m-section and an a=ssrc-group:FEC-FR <media-ssrc> <fec-ssrc></code> line.</td> Parity bytes are wrapped inside a RED packet that uses the same SSRC as the media. No extra m-section needed; you just add a RED and a ULPFEC payload-type.</td></tr> Codecs</strong></td> True Codec-agnostic</strong>, works with VP8/9, H.264, AV1 … anything that produces RTP packets.</td> While codec-agnostic</strong> in theory, Chromium limits full protection to VPx and AV1. With H264 and H265 the encoder can't flag per-partition loss importance in a way that is useful for FEC. ULPFEC + Nack for codecs without Picture-ID will have to retransmit FEC packets, So ULPFEC is not used for H264 and H265.</td></tr> Loss patterns handled</strong></td> Two-dimensional (rows × columns) parity</td> Single-direction parity</td></tr> Implementation status</strong></td> All Chromium-based browsers can receive</em> FlexFEC; send-side</strong> support is still immature. Limited receive support in Safari and Firefox doesn't support it.</td> Send + Receive</strong> in Chrome, Firefox, Safari.</td></tr> </tbody></table> FlexFEC is the only option that remains fully codec-agnostic. ULPFEC does a fine job for VP8/VP9/AV1, but offers little benefit for H264 or H265.</p> Codec-integrated FEC</a></h3> While FlexFEC and ULPFEC live at the RTP packet layer, some codecs integrate redundancy straight into the encoded payload itself.</p> The only such codec you'll meet in browsers today is Opus, and the way it works is when you set useinbandfec=1</code>, the Opus encoder embeds a low-bit-rate redundancy (LBRR) copy of frame N-1 inside the packet that carries frame N. If packet N-1 is lost, the decoder can reconstruct it from the tail of N.</p> FEC isn't magic: trade-offs to keep in mind</a></h2> Bandwidth overhead</strong> If you add 20% FEC without raising the send cap, you'll need to reduce the video resolution or quality to stay within the cap. In good network conditions where the packet loss is minimal, FEC steals bandwidth in exchange for redundancy you don't need.</p> </li> Extra load on a congested link</strong> When the loss is caused by congestion, adding more packets for FEC can deepen the problem, in congested networks, sometimes disabling FEC yields better results.</p> </li> Loss pattern mismatch</strong> Random 1-2% loss is where light FEC shines. But for example, 10% bursts every few hundred ms requires high parity, which often ends up worse than just using RTX.</p> </li> Silent failures</strong> Analytics may count frames as "recovered" even if the end user gets artifacts or glitches.</p> </li> Only packets, not outages</strong> A 100ms network stall drops all packets, media and parity alike; so buffering or retransmission is still needed.</p> </li> FEC is best for high-latency networks</strong> FEC is designed to avoid the delay of waiting for retransmissions, but for example, if your round-trip time (RTT) is under 30-50ms, do you really need it? In low-latency networks, simply retransmitting lost packets might be more efficient and better overall than sending redundant data up front.</p> </li> </ol> So, start with the lowest cost protection first (Opus in-band FEC for audio, RTX for video). Add FEC only when telemetry shows sustained loss or when you have bandwidth room. FEC should be a dynamic option, not a set-and-forget checkbox that you enable for all users, and all network conditions.</p> FEC algorithms beyond XOR</a></h2> FlexFEC and ULPFEC both use XOR-based recovery logic to generate packets. Another family of algorithms, like Reed-Solomon</a>, can recover from more complex losses but are currently not standardized in WebRTC due to their higher computational cost and block-based design, which is not best suited for real-time use in WebRTC.</p> If you're curious, the klauspost/reedsolomon Go library</a> offers a SIMD-accelerated implementation that's fun to experiment with.</p> FlexFEC in Pion</a></h2> Pion now supports basic FlexFEC encoding! You can pair it with Chromium and Safari (receive-only), and it's simple to integrate – just register the payload type in your media engine and add the interceptor, then you're good to go! More about it in Enable FlexFEC Encoding in Pion</a>.</p> For a quick demo, you can check out the play-from-disk-fec</a> example.</p> What to look forward to in the final FlexFEC implementation</a></h3> If you worked with FlexFEC, you may have come across multiple draft versions, FlexFEC-03</strong> and FlexFEC-20</strong>. The final standard is published as RFC 8627</a>, and while it builds on ideas introduced in the -20 draft, it formalizes several features and adds important clarifications.</p> Although Pion doesn't yet fully support RFC 8627, we're tracking it closely and excited about the improvements it will bring, here are some of the highlights:</p> Flexible Masking (Selective Protection)</strong>: RFC 8627 introduces a flexible mask</strong> mode, allowing senders to protect an arbitrary subset of source packets. Instead of a fixed pattern, it sends a bitmask indicating exactly which packets are covered. This can be used to implement smarter FEC strategies, such as protecting only keyframes or higher-priority codec configuration packets. -> RFC 8627, Section 1.1.4</a></p> </li> Hybrid FEC and Retransmission Support</strong> The final spec explicitly supports both proactive FEC</strong> and RTP retransmission</strong>. It also requires that when FlexFEC is used alongside another retransmission mechanism, the answer in an SDP negotiation must use FlexFEC only, ensuring consistent protection handling. -> RFC 8627, Section 1.1.7</a></p> </li> Improved Scalability and Consistency</strong> RFC 8627 addresses scalability limitations from earlier FEC mechanisms like RFC 5109</a> and RFC 2733</a>. It also reuses the RTP repair stream format</strong> defined in RFC 6363</a>, making it more consistent with modern RTP practices and easier to interoperate with existing tooling. -> RFC 8627, Section 1.1.1</a></p> </li> Packet and Header Format Clarifications</strong> The final spec clearly defines key header fields. For example, the R bit</strong> distinguishes retransmission (R=1) from FEC repair (R=0), while the F bit</strong> selects between flexible-mask mode (F=0) and fixed-offset mode (F=1). These explicit semantics simplify implementation logic and parsing. -> RFC 8627, Section 6.2</a></p> </li> </ol> Enable FlexFEC Encoding in Pion</a></h3> You can use the helper for quick setup:</p> err</span> =</span> webrtc</span>.</span>ConfigureFlexFEC03</span>(</span>fecPayloadType</span>,</span> mediaEngine</span>,</span> interceptorRegistry</span>)</span></span></code></pre> Or setup codec and interceptor by hand:</p> Register FlexFEC03 codec to MediaEngine</li> </ol> err</span> =</span> mediaEngine</span>.</span>RegisterCodec</span>(</span></span> webrtc</span>.</span>RTPCodecParameters</span>{</span></span> RTPCodecCapability</span>:</span> webrtc</span>.</span>RTPCodecCapability</span>{</span></span> MimeType</span>:</span> webrtc</span>.</span>MimeTypeFlexFEC03</span>,</span></span> ClockRate</span>:</span> 90000</span>,</span></span> Channels</span>:</span> 0</span>,</span></span> SDPFmtpLine</span>:</span> "</span>repair-window=10000000</span>"</span>,</span></span> RTCPFeedback</span>:</span> nil</span>,</span></span> }</span>,</span></span> PayloadType</span>:</span> 118</span>,</span> //</span> You can change this</span></span> }</span>,</span></span> webrtc</span>.</span>RTPCodecTypeVideo</span>,</span></span> )</span></span></code></pre> Register FlexFEC interceptor</li> </ol> fecInterceptor</span>,</span> err</span> :=</span> flexfec</span>.</span>NewFecInterceptor</span>(</span>)</span></span> //</span> handle error...</span></span> interceptorRegistry</span>.</span>Add</span>(</span>fecInterceptor</span>)</span></span></code></pre> That's it! But with a few catches:</p> The order of the FlexFEC interceptor in the chain of interceptors matters. It should be registered before TWCC header extension interceptor</a> and NACK responder interceptor</a> if you are using these.</li> If you are using congestion controller, be aware of the bandwidth FEC is taking, you should set your encoder's bitrate to cc_estimated_bandwidth - fec_bitrate</code>. By default, for every 5 media packets, 2 FEC packets will be produced, so expect roughly 40% bitrate overhead. You can tune FEC parameters to reduce or increase it.</li> Enabling FlexFEC encoding may introduce significant CPU/RAM overhead.</li> Current implementation does not protect media packet batches if they contain missing or reordered packets. So, it works best with senders that packetize media themselves.</li> </ol> Configuring FlexFEC Encoder Interceptor</a></h3> You can customize FEC protection by tweaking NumMediaPackets</code> and NumFECPackets</code> using the interceptor's options</a>.</p> It will result in a protection profile which uses NumFECPackets</code> FEC packets to protect each NumMediaPackets</code> media packets. Interleaved protection will be used, which means that media packet X</code> will be protected by FEC packet (X mod NumFECPackets)</code>.</p> Verify that FlexFEC is Enabled</a></h3> On sender side, which is within Pion, you can log the packets using the packetdump interceptor</a>.</li> On receiver side, for example, in Chromium, you can go to chrome://webrtc-internals</code> and search for the stats graph of fecBytesReceived</code>.</li> </ul> Credits</a></h2> Huge thanks to everyone who contributed to this project: Aleksandr Alekseev</a>, Jingyang Kang</a>, Jo Turk</a>, Thomas Pouget-Abadie</a>.</p> Pion Opus Sean DuBois — Sun, 02 Oct 2022 00:00:00 +0000 I am so excited to announce that pion/opus</a> is now available. It is a Pure Go implementation of the Opus Codec</a>. pion/opus is far from done though. It is far enough along for others to start playing with and contributing too though. We still have a lot more work to do. If you are interested now is a great time to get involved!</p> Now that the code is shared I wanted to share my goals, what I learned and hopes for the future.</p> Why Opus?</a></h3> I had a few a different reasons for starting this project. Some of the goals I had from the very begining, others I picked up along the way.</p> Complete the WebRTC Stack</strong> - Implementing Opus was a clear next step for the Pion project. For the last 5 years we have been developing different parts of the WebRTC Stack</a>. We have completed everything but audio and video codecs. It feels great getting closer to the goal of having the entire software stack in one memory.</p> Empower others to learn Opus</strong> - When implementing pion/opus I designed it so others can learn from it. silk/decoder.go</a> includes the relevant comments from the RFC. I made sure to use the same variable names and flow as the RFC. My hope is that developers can jump into the code and learn quickly from the unit tests and code that follows the RFC.</p> Enable Pion users</strong> - Pion users have been building more and more interesting things. One developer is building Go clients that make it easier for musicians to play together. Another developer is building clients that make it easier for his customers to bridge protocols. The last developer is building a WebRTC/Phone bridge. All of them have talked about frustrations with cross compiling and making C libs work with Go. I am sure more developers have been frustrated and just haven't felt comfortable enough to discuss it.</p> More efficent use of Opus</strong> - I hope to expose the internals of this library so users can more efficently work with Opus. Maybe instead of running a Fast Fourier transform to implement a talking notifier developers will only do partial decode. I would also like to build a bitstream analyzer using this library to help users trying to learn Opus. I am not sure what will happen, but it is always suprising what developers build when you give them more tools.</p> Proving Myself</strong> - Codecs were always a daunting topic to me. I spend a lot of time using codecs, but never really understanding them. It felt like a chance for growth to take this on. I also get asked lots of questions about coded media and have to answer in generalizations. I hope in the future I can give people more concrete answers after this project is complete.</p> Inspire more Go media projects</strong> - Go is a incredibly powerful language. You can build scalable, secure and maintainable software quickly. If you have the libraries you need. I believe having a Opus implementation will inspire other developers to join in the challenge of making Go better for media development. I want a future with more and more Go media projects, that can't happen unless we write these libraries.</p> Get a new group of developers involved in Pion</strong> - Pion has primarily been WebRTC users. I hope that by having a Opus implementation it will bring a new set of eyes on the Pion project. I think all the existing Pion projects would benefit greatly from developers joining with different opinions and backgrounds. Personally it will be motivating to work with new people who have new demands. The pressure of Pion has lowered lately as the core libraries have stabilized.</p> Create opportunity for contributors</strong> - My favorite part of working on Pion is seeing the opportunities it creates. It brings me a lot of happiness watching developers grow as they contribute to Pion. They become better programmers and collaborators. Pion also provides lots of great career opportunities. I have see multiple contributors get high paying jobs because of the skills developed while working on Pion. It has ranged from new developers who haven't finished college to seasoned developers with decades of experience. Getting involved in Open Source can be rewarding for anyone.</p> What I learned</a></h3> Not as much about audio as I hoped! When implementing this codec I spent a lot of time reading RFC6716</a>. The RFC talks about the how, and not the why though. It is possible to learn the high level details like LPC</a>. I wasn't able to find any details about the particulars of SILK itself. When this project is done I hope to go research that and write it down.</p> A few implementations of Opus exist. libopus</a>, concentus</a>, lu-zero/opus</a> and FFMpeg</a>. If you want to learn Opus I found the Rust implementation and FFmpeg the easiest to understand. It is surprising how most of these efforts were driven by a single person!</p> The Opus community is very helpful. A special shout out to Luca Barbato who responded to many emails from me. Without his help I would have never made it this far. The developers in the #opus libera channel were very helpful also. Timothy Terriberry took a lot of time answering my questions on IRC. He also provided earlier/simpler implementations of Silk which helped me understand things a lot better.</p> Future</a></h3> I don't know how much actual usage this project will see. I think I could have spent my time better elsewhere, but I am happy that I didn't quit. Quite a few times I wanted to give up. I held on to the idea that it is better to finish late and have something to show, then quit and have nothing at all.</p> My prediction/hope is that this project will give me an opportunity to work with smart and interesting people. I anticipate that I will get some contributors that want to learn. I also think this project will mean that webrtc-rs</a> will get a Opus implementation. With this and rav1e</a> I see a future for Pure Rust WebRTC clients!</p> ION, grown up and moving out. Sean DuBois — Tue, 19 Jul 2022 00:00:00 +0000 Going forward the ION project will now be hosted at ionorg/ion</a>. ION itself is not changing in any other way. We are just making these changes to more clearly reflect the actual leadership and organization of the project. Changes like this aren't free though. Users may have to change imports, tooling and other software. We apologize for that, but hope these benefits are worth it.</p> Recognize the actual leadership of ION</a></h3> The leadership and contributors of ION and Pion are often conflated. Often I will have users ask me questions or even thank me for ION. This confusion is demotivating for the people actually doing the hard work. I worry that long term it will hurt the project.</p> ION was created by adwpc</a> and CloudWebRTC</a> in March of 2019. As the project has progressed it saw significant changes from tarrencev</a> and OrlandoCo</a> who made architectural changes and other leadership. ION also has a healthy community of developers answering questions and building tools. TannerGabriel</a>, leewardbound</a> and billylindeman</a> always seems to be helping users on Slack</a></p> I hope by separating ION and Pion it will make the leadership clearer and help users reach the right people.</p> Support all the Servers/SFUs written with Pion</a></h3> At the time no other Pion based servers existed. Pion users were requesting an easier solution for building servers, so it made sense to make one. As time went on developers outside the Pion project built these new and interesting WebRTC server projects (sorted alphabetically).</p> DuckSoup</a> - Videoconferencing tool for social experiments.</li> Galène</a> - SFU that is easy to deploy with low resource requirements designed for lectures, conferences and tutorials.</li> Harmony</a> - Voice and text communications platform</li> Kraken</a> - SFU designed especially for group audio conferencing.</li> LiveKit</a> - Room based API that is horizontally-scalable out of the box.</li> Plug-N-Meet</a> - Providers higher level WebRTC tools with plugins for Joomla, Moodle, Wordpress</li> </ul> With ION existing under the Pion organization users would unfairly ignore these other projects. I want to celebrate and encourage everyone who is building things with Pion.</p> Pion is a nonprofit project Sean DuBois — Mon, 27 Dec 2021 00:00:00 +0000 Pion saw its first commit Wed Mar 7 2018. I didn't have grand aspirations for the project. It felt important</a> at the time to make WebRTC more accessible. Proprietary alternatives were being proposed and they didn't have technical advantages, they were just easier to use. I enjoyed solving issues one at a time and helping users use Pion.</p> Interest in WebRTC then exploded in 2020. Lots of new demand for WebRTC software popped up over night. New developers started learning it, companies were founded around it and lots of VC money started entering the space.</p> I was approached by a few VCs who asked if I had any plans to do something commercial with Pion. It sounded exciting and I would have been fun to work on it full-time. In the end I decided that it may benefit me personally, but would be a net negative. These are the reasons I decided against it.</p> I have seen many for profit projects not have these issues. I just don't believe I have the skills/enviroment/team to make it work.</p> Reduce opportunities for individuals</a></h3> Working on Open Source early in my career had a huge impact on me, and I hope that working on Pion can do the same for others. A few milestones that mean a lot to me.</p> First job in video because of a project they built with Pion.</li> First speaking opportunity talking about a Pion contribution.</li> Company that uses Pion hiring multiple contributors and offering much better salaries.</li> A contributor that is an expert in RTC returning to the field through Pion</li> Companies created by first time founders using Pion</li> </ul> If Pion was for profit I don't think I could create an environment where this would all be happening. I can focus on giving individuals the best outcome and not worry about a business.</p> Harder to be a community</a></h3> For me Pion isn't just software, but a community to build things with others who are passionate. I love seeing all the projects that people share in Slack</a>. I remember lots of late night debugging sessions and difficult bugs overcome. I remember the excitement when WebRTC for the Curious</a> first started.</p> I don't think those relationships would have been made if Pion was a for profit project. I would have to prioritize issues effecting the business. I want to create an environment where developers can pour their passion into problems that interest them.</p> Competition with contributors and users</a></h3> I enjoy having an open discourse with the companies and projects that use Pion. I love learning about what people are building and learning from them. I am filled with so much joy seeing all the different use cases.</p> Robotics</a> - Makes it possible to build robots that make hospitals safer.</li> Cloud Ready WebRTC</a> - Making it so any company to build and scale WebRTC</li> Low/No-Code WebRTC</a> - Empowering the next generation of builders so they don't need to learn the details.</li> Application Streaming</a> - Users who can't afford the latest hardware can still access everything.</li> Collaborative Gaming</a> - Multiplayer NES in the browser, don't let distance stop you from spending time together.</li> P2P CDN</a> - Reduce network strain by sharing what you have already downloaded. Bringing P2P to everyone!</li> Digital Signal Processing</a> - Real time translations. Allow users to communicate with no shared language.</li> Computer Vision</a> - Process gestures and expressions. Make video conferencing feel more natural.</li> </ul> All these projects have caused Pion to grow in different ways. They all have different needs and pushed Pion in lots of good ways. If I started building competing software I would lose out on learning all these stories. Also lots of great Open Source projects Using Pion</a> that I have learned from.</p> Less diverse contributions</a></h3> If I worked on Pion full-time more issues would be addressed by me directly. I am not sure this would be better for the project long term. Pion’s growth and quality is because we have so many contributors who bring a wide variety of opinions and experiences.</p> Things like WASM Support, ORTC and Interceptors came from designs by other developers. If I had done everything myself I don't believe it would be nearly as good.</p> Divide between employees/community</a></h3> All community members are equal in the Pion ecosystem. Each developer only needs one approval to merge their PR. I don't ever want there to be a split. Employees of 'Pion Inc' would be viewed differently and would have access to resources others wouldn't have.</p> It is important to me that Pion is welcoming and inclusive to all.</p> Why WebRTC? Sean DuBois — Sun, 05 Dec 2021 00:00:00 +0000 This is a question I get in lots of different forms. Why should I use WebRTC instead of my preferred video protocol? Why does WebRTC need to be in my browser? Why does WebRTC have to be so complicated? Why do you continue to work on WebRTC projects?</p> These are my answers to the Why's of WebRTC. Lots of nuance exists, and these answers are heavily influenced by the work I do. I have been working with WebRTC since 2013. In that time I worked on two implementations Pion</a> and KVS WebRTC</a>. I also co-authored a book WebRTC for the Curious</a>. I have used WebRTC at startups</a> and corporate</a> projects.</p> WebRTC supports more then just your use case</a></h3> WebRTC supports a diverse set of use cases. It is hard to appreciate all of them when you are focused on your problem.</p> Multiple tracks of bi-directional audio and video</li> Media negotiation, codecs and media events like adding and removing tracks</li> P2P connectivity with optimal path discovery, Client/Server can be built as well.</li> Binary and text communication, lossy and unordered data with flow control APIs</li> Mandatory encryption, secure by default and requires no developer configuration</li> </ul> Supporting a wide set of features has some big benefits. Most WebRTC developers only care about a specific use case, and improve WebRTC for their needs. However, they coincidentally improve it for everyone. Lots of money has been invested in WebRTC by conferencing companies. These changes have empowered Open Source project, startups and others that don't have the same resources. Projects like Snowflake</a> are only possible because others pushed WebRTC to it's present day quality.</p> WebRTC is a standard</a></h3> WebRTC is a IETF/W3C standard. Multiple individuals have been involved since its creation. Beyond that WebRTC is made up of multiple existing standards. The majority of what WebRTC defines is how to combine these existing complicated subsystems.</p> I use WebRTC because I know it has a safe future. Protocols that are owned by a single corporation or project can be changed at any time. As an individual you also have a chance to influence WebRTC. You can get involved in the standards or write your own WebRTC software and tools.</p> WebRTC has multiple implementations and is in the browser</a></h3> WebRTC is available in every major browser and most major languages. WebRTC isn't just a browser technology!</p> C#</a></p> </li> C++</a></p> </li> C++</a></p> </li> C++</a></p> </li> C</a></p> </li> Elixir</a></p> </li> GStreamer</a></p> </li> Go</a></p> </li> Node.js</a></p> </li> Python</a></p> </li> Rust</a></p> </li> Typescript</a></p> </li> </ul> You also have a wealth of servers to choose from</p> ant media server</a></p> </li> galène</a></p> </li> ion-sfu</a></p> </li> janus-gateway</a></p> </li> jitsi</a></p> </li> kurento</a></p> </li> licode</a></p> </li> livekit</a></p> </li> mediasoup</a></p> </li> medooze</a></p> </li> Red5 Media Server</a></p> </li> </ul> WebRTC's wide availability makes it easy to build with. Developers can use the language, paradigm and tools they are most comfortable with. Just worry about learning WebRTC and not a new language. WebRTC also has a high bus factor. If a single developer or company stops supporting their software you have lots of alternatives.</p> WebRTC supports all skill levels</a></h3> You can build a simple 1:1 audio+video communication application in 50 lines of Javascript. You don't need any knowledge of VoIP or networking. WebRTC also allows a great degree of flexibility if you have more niche needs. You can ship your own Congestion Control and Error Correction for media. You can choose your own algorithm for path selection when establishing the connection. Even if you make these changes you will still be standards compliant and can connect to existing WebRTC clients!</p> WebRTC's accessibility allows a wide range of applications to be built with it. You don't need to be a video and networking expert to build your cool idea. I continue to stay involved with WebRTC because it has a rich ecosystem of developers. Life becomes stale if you build the same thing over and over again. With WebRTC I have had a chance to interact with so many different projects. Remote surgery, drones, robotics, file sharing, one hundred thousand viewer events, sub-second bidding for cattle, game streaming, VPNs in the browser, real-time voice translation, and these are just the public projects! WebRTC lets beginners bring their interesting ideas to life.</p> DTLS 1.3 Is Coming to Go: An Implementer's Perspective Theodor Midtlien — Thu, 01 Jan 1970 00:00:00 +0000 What Is DTLS?</a></h2> DTLS is the Datagram (UDP) variant of TLS. You might not be familiar with TLS, but you use it extensively!</p> TLS is the protocol used for secure communication across the Internet (and intranets). Most web browsing and many email and VPN services run over it.</p> (D)TLS solves three things:</p> Encryption</strong> - Prevent attackers from seeing the protected data</li> Authentication</strong> - Confirm who you are communicating with</li> Integrity</strong> - Detect tampered or corrupted records</li> </ul> To accomplish this, TLS standardized a handshake so two TLS peers can agree on a cipher suite and keying material. DTLS extends and modifies the TLS handshake so it can work over a lossy protocol.</p> If you are curious about deeper details on the protocol, see WebRTC for the Curious - Securing</a>.</p> DTLS 1.3 Upgrades</a></h2> DTLS 1.3 had quite a few changes. These are the ones we noticed and found most exciting.</em></p> Uses Less Data</a></h3> Each DTLS 1.2 record has a header with metadata required for the session. Each record uses a fixed 13-byte</strong> record header:</p> 0 1 2 3 4 5 6 7</span></span> +-+-+-+-+-+-+-+-+</span></span> \| Content Type \| 1 byte</span></span> +-+-+-+-+-+-+-+-+</span></span> \| Version \| 2 bytes</span></span> + +</span></span> \| \|</span></span> +-+-+-+-+-+-+-+-+</span></span> \| Epoch \| 2 bytes</span></span> + +</span></span> \| \|</span></span> +-+-+-+-+-+-+-+-+</span></span> \| \|</span></span> + +</span></span> \| \|</span></span> + Sequence + 6 bytes</span></span> \| Number \|</span></span> + +</span></span> \| \|</span></span> +-+-+-+-+-+-+-+-+</span></span> \| Length \| 2 bytes</span></span> + +</span></span> \| \|</span></span> +-+-+-+-+-+-+-+-+</span></span></code></pre> DTLS 1.3 uses a variable-length unified header that can be as small as 2 bytes</strong>:</p> 0 1 2 3 4 5 6 7</span></span> +-+-+-+-+-+-+-+-+</span></span> \|0\|0\|1\|C\|S\|L\|E E\| 1 byte</span></span> +-+-+-+-+-+-+-+-+</span></span> \| Connection ID \| optional, variable length</span></span> \| (if any) \|</span></span> +-+-+-+-+-+-+-+-+</span></span> \| 8- or 16-bit \| 1 or 2 bytes</span></span> \|Sequence Number\|</span></span> +-+-+-+-+-+-+-+-+</span></span> \| 16-bit Length \| optional, 2 bytes</span></span> +-+-+-+-+-+-+-+-+</span></span> </span> C = Connection ID present</span></span> S = Sequence number length</span></span> L = Length present</span></span> E = Epoch</span></span></code></pre> With Pion's default MTU of 1200, you could save 50 MB</strong> on a 5 GiB transfer!</strong> That might not sound like a lot, but at scale, it will add up to massive savings.</p> </blockquote> Connects Faster</a></h3> DTLS 1.3 connects faster for two reasons.</p> First, DTLS 1.3 has explicit ACKs.</strong> During the handshake, if a datagram was lost in DTLS 1.2, both sides would wait and then retransmit on timeout. With DTLS 1.3, each side can send explicit ACK</code> messages. This allows selective or early retransmission after a datagram loss.</p> Second, DTLS 1.3 allows a key share in the ClientHello</code>.</strong> This removes an entire round trip. The server no longer waits for the ClientKeyExchange</code>!</p> This allows the handshake to be shrunk down to only 1 RTT</strong> (from 2</strong> in DTLS 1.2).</p> </blockquote> Below are simplified versions of the handshakes. Note how the client only has to wait for one server flight to receive encrypted data (Application Data) in DTLS 1.3, versus waiting for two server flights in DTLS 1.2.</p> DTLS 1.2</a></h4> Client Server</span></span> ------ ------</span></span> Flight 1</span></span> ClientHello --------></span></span> </span> Flight 2</span></span> <-------- ServerHello, ServerHelloDone</span></span> </span> Flight 3</span></span> ClientKeyExchange, --------></span></span> ChangeCipherSpec, Finished</span></span> </span> Flight 4</span></span> <-------- ChangeCipherSpec, Finished, Application Data</span></span> </span> Server data after 2 RTTs</span></span></code></pre>DTLS 1.3</a></h4> Client Server</span></span> ------ ------</span></span> Flight 1</span></span> ClientHello + key_share -----></span></span> </span> Flight 2</span></span> <----- ServerHello, EncryptedExtensions, Finished, Application Data</span></span> </span> Server data after 1 RTT</span></span></code></pre>Improved Security</a></h3> DTLS 1.3 improves security in a bunch of ways: lots of insecure and brittle stuff was removed, and a few new things were added.</p> Renegotiation was removed.</strong> Renegotiation was a source of bugs and security issues (CVE-2009-3555</code>) and was entirely removed. Pion DTLS never implemented renegotiation, but we are excited not to get requests for it anymore!</p> More of the handshake is encrypted.</strong> A big improvement is that certificates are no longer exchanged in plaintext. This was a big fingerprinting surface that is now closed.</p> Fragile and difficult-to-implement cipher suites are removed.</strong> Only AEAD</code> cipher suites are available now. AEAD</code> is easier to use because encryption and authentication are done in one call.</p> Compare Pion's use of CBC</a> to its use of AEAD</a>.</p> With CBC</code>, we have to do authentication and encryption separately. AEAD</code> combines authentication and encryption into one operation. Another benefit is that AES-GCM</code> is often more performant. Since CBC-and-HMAC</code> involves two operations, you end up with more copying and instructions.</p> Forward secrecy is required for all non-pre-shared-key sessions.</strong></p> In DTLS 1.2, you had the option to use cipher suites with RSA</code> key exchange or ephemeral key exchange. Ephemeral key exchange means you generate a new key for each session. RSA means the server would reuse the same key across many sessions. If an attacker got access to the private key for RSA sessions, they could decrypt every session. With ephemeral keys, the attacker needs the keys for each individual session.</p> A pre-shared-key session can still be configured not to use ephemeral keys. Devices that are battery-powered or constrained might not be powerful enough to generate an ephemeral key pair for each session, so they may use this instead.</p> Connection Identification Added</a></h3> One of the exciting things about UDP is its connectionless nature. A user can stream media or upload a file and switch networks without interruption.</p> This wasn't easily possible with DTLS because it doesn't provide a session identifier. Most deployments would depend on the client's IP address and port as the session identifier.</p> DTLS 1.3 and RFC 9146, an extension to DTLS 1.2, have a fix for this, though! DTLS record headers can now contain a unique ID for the session.</p> You can now uniquely identify your DTLS traffic. This solves a few problems for DTLS users.</p> Identify clients</strong> as they switch between Wi-Fi and cellular networks.</li> Load balance effectively</strong> by routing sessions to specific servers.</li> Support long-running sessions.</strong> An IoT device can shut down and come back days later.</li> </ul> Implementer's Perspective</a></h2> Finished Is a Different Story in 1.3</a></h3> In DTLS 1.2, you have one finite state machine. After the handshake finishes, you are done.</strong> With DTLS 1.3, things get a bit more complicated! You need to handle NewSessionTicket</code> and KeyUpdate</code>. Going into this project, I didn't appreciate that complication from just reading the IETF doc.</em></p> This required a second state machine</strong> with its own ACK</code> handling, retransmission timer, etc. See Pion's post-handshake implementation</a>.</p> — Sean-Der</a></em></p> The Names Are Scarier Than the Concepts</a></h3> The TLS and DTLS specs make things feel really complex, but the concepts underneath are actually pretty simple. The spec is dense with terms like HKDF</code>, AEAD</code>, and AES</code>. Once you learn the vocabulary, it's actually quite easy to follow!</p> — Jo Turk</a></em></p> The Spec Isn't Linear</a></h3> You can implement something near the end of the spec and discover that you need to refactor everything from the record layer to the ciphers. Section 6.1</a> introduces epochs, but it isn't until Section 8</a> that it becomes clear that the client can be at epoch 3 while the server is at epoch 4, and that the state needs to be explicitly directional.</p> Our initial implementation maintained only one active epoch and record protection per direction. Supporting KeyUpdate</code> required per-epoch state and a series of refactors. See the tracking issue</a>.</p> — Jo Turk</a></em></p> The Importance of Interop Testing</a></h3> While working on Pion DTLS, Jo Turk</a> created a dedicated repository for DTLS interoperability testing: dtls-interop</a>.</p> Testing against wolfSSL she found three issues in Pion's DTLS 1.3 implementation (and fixed them)</p> Recognize partial handshake retransmissions</a></li> Resume and ACK split flights</a></li> Handle duplicated Flight 4</a></li> </ul> It currently tests against OpenSSL, BoringSSL (Chrome), and wolfSSL. Firefox and more will be added soon!</p>

Fixing WebRTC data-channels head-of-line blocking with RFC-8260

Jo Turk — Sun, 17 May 2026 00:00:00 +0000

What is head-of-line blocking and why it's an issue with WebRTC datachannels</a></h2>

SCTP (the protocol used by WebRTC data channels) looks like it should be perfect for multiplexed application data. It is message-based, it supports multiple streams inside one association, and WebRTC data channels are built on top of it for non-media data.</p>

To understand sender-side head-of-line blocking, we first need to understand how SCTP sends messages. Messages are scheduled based on their send order. SCTP fragments messages according to the path MTU (Maximum Transmission Unit), which is usually around 1200 bytes for WebRTC. A 256 KB message would therefore be split into roughly 218 smaller fragments (chunks).</p>

flowchart LR</span></span>
    AppBulk["App sends large message"] --> SCTP["Classic SCTP sender"]</span></span>
    AppControl["App sends tiny small message"] --> SCTP</span></span>
</span>
    SCTP --> Frag["Fragment large message<br/>into DATA chunks"]</span></span>
    Frag --> Queue["large fragments stay together"]</span></span>
    Queue --> Wire["Wire order"]</span></span>
</span>
    Wire --> F0["large frag 0"]</span></span>
    F0 --> F1["large frag 1"]</span></span>
    F1 --> F2["large frag 2"]</span></span>
    F2 --> FMore["... many more fragments ..."]</span></span>
    FMore --> tiny["tiny small message"]</span></span></code></pre>So what is the actual problem?</a></h3>
Imagine you're streaming media over SCTP for some special use case, maybe DRM protected-content or codecs that aren't normally supported by WebRTC media tracks (RTP), such as AAC audio or VVC video. At the same time you have control, telemetry or chat streams sharing the same SCTP association.</p>
If a media stream starts sending large reliable messages, SCTP will fragment them into smaller chunks, and those fragments can monopolize the send queue. In other words, small chat or control messages may end up waiting behind the large transfer.</p>
If the stream is reliable and packets are lost, retransmissions of those large fragments can delay smaller streams even further. Chat messages will look delayed, and control traffic will feel laggy.</p>
large video fragment 0</span></span>
large video fragment 1</span></span>
large video fragment 2</span></span>
large video fragment 3</span></span>
...</span></span>
large video fragment 200</span></span>
tiny control message finally sent</span></span>
chat messages finally sent</span></span>
audio-like frame finally sent</span></span></code></pre>So why can't I just fragment the messages myself and have my own queue?</a></h3>
Many applications do exactly that to avoid large SCTP messages, instead of sending one huge message, the application splits it into smaller chunks (just like SCTP itself), and schedules those chunks itself.</p>
But the tradeoff is that you're now effectively building another protocol on top of SCTP.</p>
And you need to handle things like chunk numbering, reassembly, flow control, backpressure handling, retransmission semantics, fair scheduling, and message boundaries.</p>
This gets trickier depending on the stream mode (pick your poison).</p>
If you use partially reliable streams or unreliable streams (SCTP won't retransmit lost packets), you need to decide what happens when fragments are dropped, should it be discarded, partially decoded, retransmitted, some sort of FEC</a>?</p>
But if you decide you're not going to deal with a complex unreliable handling, and you use reliable streams, then you're back dealing with delayed delivery on other streams under packet loss and retransmissions on bad networks.</p>
RFC 8260</a> was written to fix exactly this class of problem. The issue was that TSN (Transmission Sequence Number), was doing too many jobs at once: reliability, fragment reassembly, and sequencing. fragmented messages also had to use consecutive TSNs.</p>
What I-DATA changes</a></h2>
SCTP message interleaving uses the I-DATA</strong> chunk. The important change is that TSN is no longer used to order fragments inside a user message. I-DATA adds:</p>
MID = Message Identifier</span></span>
FSN = Fragment Sequence Number</span></span></code></pre>
The TSN still exists</em>, though. It is still used for reliability, SACKs (Selective Acknowledgment), loss detection, and retransmission. But fragment reassembly now uses MID + FSN</strong>, not "all fragments must be adjacent in TSN space." RFC 8260 says I-DATA adds MID and FSN, removes SSN, uses MID to identify the message, and uses FSN to enumerate fragments of that message.</p>
So the identity becomes:</p>
TSN = reliability / SACK / retransmission</span></span>
SID = SCTP stream</span></span>
MID = user message inside that stream</span></span>
FSN = fragment number inside that message</span></span></code></pre>
Example:</p>
SID=4 MID=90 FSN=0 TSN=100  video fragment</span></span>
SID=2 MID=301 FSN=0 TSN=101 chat message</span></span>
SID=0 MID=12 FSN=0 TSN=102 control message</span></span>
SID=4 MID=90 FSN=1 TSN=103 video fragment</span></span>
SID=1 MID=44 FSN=0 TSN=104 chat message</span></span>
SID=4 MID=90 FSN=2 TSN=105 video/ fragment</span></span></code></pre>
The video fragments still reassemble correctly because the fragments are identified by:</p>
SID=4 + MID=90 + FSN=0,1,2,...</span></span></code></pre>
They no longer need to occupy one contiguous TSN range :)</p>
Interleaving under loss</a></h2>
Interleaving does not make packet loss disappear. It does not give every stream its own congestion window. It does not magically create bandwidth, It just lets other streams compete for send opportunities between fragments of a large message.</p>
With interleaving, a bad network can still look like this:</p>
bulk fragment sent</span></span>
audio-like frame sent</span></span>
control message sent</span></span>
bulk fragment sent</span></span>
telemetry sent</span></span>
bulk fragment lost</span></span>
audio-like frame sent</span></span>
retransmit lost bulk fragment</span></span>
control message sent</span></span></code></pre>
That is much better than:</p>
bulk fragment sent</span></span>
bulk fragment sent</span></span>
bulk fragment lost</span></span>
bulk fragment sent</span></span>
bulk fragment sent</span></span>
retransmit lost bulk fragment</span></span>
control finally sent</span></span>
audio-like frame finally sent</span></span>
telemetry finally sent</span></span></code></pre>
The retransmission still costs capacity, but it no longer has to trap every other stream behind the large message’s original serialization.</p>
Where "Stream Schedulers" fit in the title</a></h2>
Interleaving gives SCTP the ability to pick chunks from different streams. But it still needs a policy for deciding which stream goes next.</p>
Common policies are:</p>
round-robin:</span></span>
  each non-empty stream gets turns</span></span>
</span>
priority:</span></span>
  high-priority streams are served before lower-priority streams</span></span>
</span>
WFQ:</span></span>
  weighted fair queuing; streams receive capacity according to weights</span></span></code></pre>
RFC 8260 defines SCTP stream schedulers, including round-robin, priority, and weighted fair queuing. It says WFQ uses configurable per-stream weights and that, if one stream has n times another stream's weight, it should receive n times the capacity. It also says WFQ with user-message interleaving is used for WebRTC data channels.</p>
For HOL-sensitive apps, this is the powerful combination:</p>
I-DATA interleaving + WFQ or priority scheduling</span></span></code></pre>
Interleaving breaks the large-message wall. WFQ or priority decides who gets protected.</p>
Pion/SCTP implementation for RFC 8260 comes with WFQ scheduler (default), WFQ with custom weight so you can prioritize a specific stream (func WithInterleavingWeightedFairqueuingWeight(streamID uint16, weight uint16)</code>), and round robin scheduler (enabled via WithInterleavingRoundRobinScheduler()</code>).</p>
and if you feel like it you can even define your own custom scheduler :)</p>
type</span> InterleavingStreamScheduler</span> interface</span> {</span></span>
	Reset</span>(</span>)</span></span>
	Push</span>(</span>StreamSchedulerChunk</span>)</span></span>
	Peek</span>(</span>)</span> StreamSchedulerChunk</span></span>
	Pop</span>(</span>StreamSchedulerChunk</span>)</span> error</span></span>
}</span></span>
</span>
func</span> WithInterleavingStreamSchedulerFactory</span>(</span>newScheduler</span> InterleavingStreamSchedulerFactory</span>)</span></span></code></pre>The interleaving-inspection example</a></h2>
I added an example</a> to pion that lets you inspect and play with interleaving, it has a basic pressure test, if you have interleaving enabled, you'll see something like this (notice all control messages arrive before bulk chunks complete)</p>
</p>
if you don't have interleaving enabled, you'll see something like this (notice how control messages had to wait for bulk chunks to complete):</p>
</p>
So how can I enable and use interleaving?</a></h2>
Interleaving should be enabled by default in Pion 4.2.13 and above, and there is nothing you need to do.</p>
As I wrote this, interleaving is enabled by default in Firefox, and is behind a flag in Chrome, and merged to Pion/sctp</a>.</p>
Interleaving isn't actually used unless it's supported by the other side, and negotiated during the handshake. To know if a current stream has interleaving or not, we expose it through Pion's GetStats()</p>
report</span> :=</span> pc</span>.</span>GetStats</span>(</span>)</span></span>
</span>
for</span> _</span>,</span> stat</span> :=</span> range</span> report</span> {</span></span>
    sctpStats</span>,</span> ok</span> :=</span> stat</span>.</span>(</span>webrtc</span>.</span>SCTPTransportStats</span>)</span></span>
    if</span> !</span>ok</span> ||</span> sctpStats</span>.</span>Metadata</span> ==</span> nil</span> {</span></span>
        continue</span></span>
    }</span></span>
</span>
    fmt</span>.</span>Println</span>(</span>"</span>SCTP interleaving:</span>"</span>,</span> sctpStats</span>.</span>Metadata</span>.</span>MessageInterleavingEnabled</span>)</span></span>
    fmt</span>.</span>Println</span>(</span>"</span>Partial reliability:</span>"</span>,</span> sctpStats</span>.</span>Metadata</span>.</span>PartialReliabilityMode</span>)</span></span>
}</span></span></code></pre>
The stats metadata includes:</p>
{</span></span>
  "</span>metadata</span>"</span>:</span> {</span></span>
    "</span>messageInterleavingEnabled</span>"</span>:</span> true</span>,</span></span>
    "</span>partialReliabilityMode</span>"</span>:</span> "</span>i-forward-tsn</span>"</span>,</span></span>
    "</span>zeroChecksumSendingEnabled</span>"</span>:</span> true</span>,</span></span>
    "</span>zeroChecksumReceivingEnabled</span>"</span>:</span> true</span></span>
  }</span></span>
}</span></span></code></pre>
We might have similar stats in the browser API if W3C accepts my issue</a>.</p>
Thanks</a></h2>
Huge thanks to:</p>

Sean DuBois</a> For introducing me to this spec, I saw him talking about it a few years ago, and for making and inviting me to Pion :)</li>
R Chiu</a> For making pion/sctp fun again. We hope to see her again in Pion.</li>
Nils Ohlmeier</a> For helping me navigate I-DATA support in Firefox, even tho I ghosted him for a bit after I got a new job :)</li>
Valor Zard</a> For making a list of all the libraries that supports RFC-8260 while I was doing interop tests, and for consistently and inderiectly reminding me to finish Pion's RFC-8260 implementation.</li>
Philipp Hancke</a> for helping me trying to get interleaving as a transport stats to w3c</a></li>
Júlia Paschoalinoto</a> and Franta</a> for all the manual testing and feedback.</li>
</ul>



RACK makes Pion SCTP 71% faster with 27% less latency
R Chiu — Sun, 21 Dec 2025 00:00:00 +0000
What is SCTP?</a></h2>
SCTP stands for Stream Control Transmission Protocol. At a basic level, SCTP is designed to be reliable, handle de-duplication of packets, and support packets that may be delivered in order or out of order. Beyond transporting messages, SCTP can also set up a connection between users. On a deeper level, SCTP includes native support for multiplexing: multiple applications can take advantage of a single transport connection. SCTP also supports multi-homing, which enables automatic failover from a primary connection to a secondary one.</p>
At the most basic level, it lets you reliably send information from one computer to another without any complications.</p>
What is SCTP used for?</a></h3>
SCTP's uses can generally fit into two cases:</p>
1. Sending some amount of data.</a></h4>
Imagine a scenario where two people are texting when one person remembers a picture that they want to send. As they text back and forth, an image gets uploaded, which takes some time to get sent. SCTP can handle multiple things going on at the same time and doesn't delay any messages from being sent just because an image is being uploaded! Thanks to SCTP, text messages can be safely delivered to each person and nothing in their conversation is lost in transit or delayed just because something else is being transferred at the same time as their messages.</p>
Building on this idea, users can share larger files with each other. This includes anything: birthday videos, audio recordings, even boring paperwork; anything that's a file can be sent!</p>
2. Sending small amounts of data with a purpose.</a></h4>
In a new scenario, imagine two people who are texting back and forth when one person gets hungry. They send a message saying, "I want a pizza!" When the other person receives the text, they think, "Maybe I should do something about that!" The recipient can choose to do something useful for the sender with that information.</p>
This is the blueprint for many awesome technologies today, as it opens up the possibility of controlling one computer from a different computer. Consider a surgeon who performs an operation involving a remote-controlled device that needs to respond with as little latency as possible. Similarly, real-time navigation systems also need to respond to changes in traffic conditions quickly in order to avoid congested or unsafe areas due to accidents or weather conditions.</p>
Other uses:</a></h4>
SCTP can be used for online multiplayer games where every frame counts, including first-person shooters and fighting games. Taking the remote surgery example in this direction leads to the idea of cloud gaming, as players can have their inputs sent to a different device than the one that they're using while still being able to play the game!</p>
SCTP is also used inside web browsers via WebRTC and has found use in AI applications and cryptocurrency-related technologies. Additionally, payment verification can similarly benefit from secure and fast communication.</p>
Why SCTP for WebRTC?</a></h2>
SCTP is used for WebRTC because of its ability to send information via reliable and unreliable datachannels. For example, you can send messages or files in a chat with SCTP. Other uses include being able to know when users toggle their microphone or video in a video call. In some special cases, SCTP can even be used to transmit video between users, but that's significantly less common.</p>
In WebRTC, the ICE protocol connects users and the DTLS protocol establishes a secure connection, at which point SCTP is then used to securely transfer data. In an ideal setup, data that's sent should "just work". Unfortunately, that isn't how things tend to pan out, as issues eventually crop up. Packets get dropped, the network jitters, the computer stutters, or the coffee machine doesn't start when you thought it had. That's why it's important to have a backup plan for when things go wrong.</p>
How SCTP Deals With Loss</a></h2>
SCTP was designed with this in mind and has two built-in recovery strategies for when networking goes wrong.</p>
The first is called "fast retransmission." The receiver detects if a chunk of data is missing in the transmission. If so, the receiver notifies the sender that a specific chunk ID is missing. If the sender receives three reports of a missing chunk where all three reports are referring to the same chunk ID, then the sender will assume that the chunk has been lost and resend it.</p>
The second is a timer-based retransmission. This happens if the receiver doesn't acknowledge that it has received all the packets within a specific window of time. If the receiver doesn't acknowledge that all the packets have been received, then the sender is prompted to retransmit the unacknowledged data.</p>
Both of these loss recovery strategies are used by SCTP to try to ensure that any lost data is detected and retransmitted as quickly as possible. At the time of writing, Pion's implementation of Pion's implementation of SCTP</a> uses these two mechanisms for loss recovery.</p>
These strategies are also used by TCP, which has prompted engineers to see if there's an even better strategy to detect and mitigate lost data.</p>
Introducing RACK</a></h2>
In February 2021, RFC 8985: The RACK-TLP Loss Detection Algorithm for TCP</a> was published. This was a completely new loss detection algorithm that focused on actively keeping track of network statistics and using timer-based signals in order to remain adaptive to ever-changing network conditions. RACK's improvements over SACK and fast retransmission in TCP were enticing enough for Linux, Windows, and FreeBSD to all implement it in TCP.</p>
While RACK was originally intended to be implemented for TCP, it is noted in the RFC that it can be implemented in other transport protocols</a>, including SCTP.</p>
The implementation for SCTP was formally analyzed in Felix Weinrank's Dissertation</a> and other publications. Weinrank's deep dive provides an extremely comprehensive review of SCTP and improvements regarding usage in various scenarios, including WebRTC. At the moment, we're more concerned with Weinrank's analysis and implementation notes regarding RACK in SCTP. In Chapter 7 of the dissertation</a>, Weinrank goes over how SCTP handles loss and how RACK can be implemented for SCTP, including extra details regarding how the implementation interacts with various SCTP extensions.</p>
RACK's motivations:</a></h2>
The authors of RACK in RFC 8985</a> provides examples of situations where RACK improves SCTP and TCP during loss recovery.</p>
How RACK's Tail Loss Probing (TLP) Works</a></h3>
sequenceDiagram</span></span>
    participant S as Sender</span></span>
    participant R as Receiver</span></span>
</span>
    Note over S,R: The two sends are normally<br>combined but are separated<br>here for visual clarity.</span></span>
    S->>R: Send 🍎</span></span>
    S-->>R: Send 🍌, 🥕, 🥔</span></span>
    Note right of R: Only 🍎 arrived, so it's ACK'd.</span></span>
    R->>S: ACK 🍎</span></span>
</span>
    Note over S,R: 2 RTTs later, TLP fires</span></span>
    S->>R: TLP triggers a retransmit 🥔</span></span>
    Note right of R: 🥔 arrived, so it's SACK'd.</span></span>
    R->>S: SACK 🥔</span></span>
</span>
    Note left of S: Mark 🍌 and 🥕 as lost</span></span>
    Note over S,R: The two sends are normally<br>combined but are separated<br>here for visual clarity.</span></span>
    S-->>R: Retransmit 🍌</span></span>
    S->>R: Retransmit 🥕</span></span>
    Note right of R: Only 🥕 arrived, so it's SACK'd<br>alongside 🥔.</span></span>
    R-->>S: SACK 🥕 and 🥔</span></span>
</span>
    Note left of S: 🍌 retransmission marked as lost.<br>Note that this is the second time<br/>that 🍌 has been marked as lost.</span></span>
    S->>R: Retransmit 🍌 again</span></span>
    Note right of R: 🍌 finally arrived!</span></span>
    R-->>S: ACK 🥔</span></span>
    Note over S,R: The ACK 🥔 is a cumulative ACK<br>for the 🍎🍌🥕🥔 sequence</span></span></code></pre>
In the above scenario, Tail Loss Probing enables the sender to quickly know if 🍎🍌🥕🥔 were successfully received. Since the sender only receives an acknowledgment (ACK) for 🍎 from the receiver, the TLP timer eventually triggers because it didn't receive an ACK for 🍌, 🥕, and 🥔. The TLP can then resend the last packet in the segment as an efficient way to:</p>


Retransmit data that the receiver would have to receive down the line anyway. The alternative would be to send an empty packet, receive an ACK, then send a missing packet. This handles both at once and can potentially save an RTT if only</em> the last packet is missing in a segment.</p>
</li>

Allow the receiver to ACK any earlier missing packets in the sequence if there were other issues due to networking, temporary stutters or freezes, etc.</p>
</li>

Check receiver responsiveness and detect if there's a network issue. Note that this is different from (2), as the receiver could potentially never respond with an ACK.</p>
</li>
</ol>
The receiver then confirms that it has 🥔 by sending a selective acknowledgment (SACK) to the sender, which tells the sender that it's received one of the packets from the segment but not all of them. At this point, the sender has an ACK for 🍎 and a SACK for 🥔, which means that it can determine that 🍌 and 🥕 must be missing. The sender then notes that 🍌 and 🥕 have been lost once. It then retransmits 🍌 and 🥕. In the example, 🍌 happens to get dropped by the network whereas 🥕 is sent and received successfully. The receiver then replies with a SACK for 🥕 and 🥔, at which point the sender can determine that 🍌 was lost a second time. Finally, the sender retransmits 🍌, which fortunately doesn't get dropped, and the receiver sends an ACK for 🥔 (note the ACK is for the last packet in the segment, which implies that all previous packets have been received).</p>
Side note: keeping track of the number of times that a packet has been lost is important as the cubic congestion control algorithm described in RFC 9438</a> (which is mentioned in RFC 8985) relies on this information.</p>
SCTP RTO vs RACK RTO</a></h3>
In this example, the authors of RFC 8985</a> show how, without RACK, SCTP and TCP can suffer from spurious retransmissions when there are retransmission timeouts (RTOs). In this case, each food icon represents an entire segment instead of a packet.</p>
sequenceDiagram</span></span>
    participant S as Sender</span></span>
    participant R as Receiver</span></span>
</span>
    S->>R: Send 🥐</span></span>
    Note over S,R: Then right before<br>the end of the RTO...</span></span>
    Note right of R: The receiver has a network hiccup!</span></span>
    S->>R: Send 🍳🥭</span></span>
</span>
    Note right of R: Received 🥐</span></span>
    Note over S,R: End of the RTO</span></span>
    R->>S: ACK 🥐</span></span></code></pre>
In this scenario, 🥐 is sent, and right before the end of the RTO, 🍳 and 🥭 are sent. The receiver gets 🥐, 🍳, and 🥭, but only manages to send an ACK for 🥐 right after the RTO. Let's see how SCTP handles this without RACK versus with RACK.</p>
sequenceDiagram</span></span>
    participant S as Sender</span></span>
    participant R as Receiver</span></span>
</span>
    S->>R: Send 🥐</span></span>
    Note over S,R: Then right before<br>the end of the RTO...</span></span>
    Note right of R: The receiver has a network hiccup!</span></span>
    S->>R: Send 🍳🥭</span></span>
</span>
    Note right of R: Received 🥐</span></span>
    Note over S,R: End of the RTO</span></span>
    R->>S: ACK 🥐</span></span>
</span>
    Note over S,R: Without RACK...</span></span>
</span>
    %% without rack</span></span>
    Note left of S: Mark 🥐🍳🥭 as lost<br>since RTO expired</span></span>
</span>
    Note right of R: Received 🍳🥭</span></span>
    Note over S,R: The Sender incorrectly<br>ignores the ack for 🍳🥭!</span></span>
</span>
    Note left of S: Prepare to retransmit 🥐🍳🥭</span></span>
    S->>R: Retransmit 🥐🍳🥭</span></span></code></pre>
We can see here that the receiver eventually sends an ACK for 🍳 and 🥭, but the sender ignores it and believes that 🥐, 🍳, and 🥭 are all missing instead of just 🥐. While it's reasonable to assume that 🥐 is missing, it's a little overzealous in retransmitting packets, which can increase network traffic during recovery, especially when it's completely possible for the sender to wait for the acknowledgments of 🍳 and 🥭 from the receiver.</p>
Let's see what RACK does!</p>
sequenceDiagram</span></span>
    participant S as Sender</span></span>
    participant R as Receiver</span></span>
</span>
    S->>R: Send 🥐</span></span>
    Note over S,R: Then right before<br>the end of the RTO...</span></span>
    Note right of R: The receiver has a network hiccup!</span></span>
    S->>R: Send 🍳🥭</span></span>
</span>
    Note right of R: Received 🥐</span></span>
    Note over S,R: End of the RTO</span></span>
    R->>S: ACK 🥐</span></span>
    Note over S,R: With RACK...</span></span>
</span>
    %% with rack</span></span>
    Note left of S: Mark 🥐 as lost<br>since RTO expired</span></span>
</span>
    Note right of R: Received 🍳🥭</span></span>
</span>
    Note left of S: Prepare to retransmit 🥐</span></span>
    S->>R: Retransmit 🥐</span></span></code></pre>
Here, RACK makes it so only 🥐 is marked as lost when the RTO expires. 🍳 and 🥭 aren't marked as lost because their own RTOs have not yet expired by the time their ACKs are received. Therefore, only 🥐 is retransmitted, as the timers for 🍳 and 🥭 would be re-armed if an ACK is received for the retransmitted 🥐.</p>
In this example, even though both non-RACK and RACK end up retransmitting 🥐 despite the receiver already having it, the focus is on minimizing spurious retransmissions. This can save on the amount of data sent over the network, which naturally speeds up any retransmissions that might occur.</p>
RACK's strategy</a></h3>
In summary, RACK's strategy generally has two main parts:</p>


Detect packet losses as quickly as possible by utilizing time-based acknowledgments of segmented data and inferences from network statistics.</p>
</li>

Use Tail Loss Probing (TLP), which sends sample data to gather more network statistics.</p>
</li>
</ol>
The combination of these two strategies allows it to quickly determine issues and properly rectify them once identified. It also provides better resilience for some tricky edge cases! If you're interested in seeing how RACK could perform in SCTP and other SCTP-specific improvements, check out chapter 7 of Felix Weinrank's thesis</a>.</p>
A quick look at the results (why this matters if you don't live in SCTP land)</a></h2>
SCP</a> is a Go test harness that runs two Pion SCTP stacks against each other inside a deterministic, in-process "virtual network" (from Pion/transport). It pins exact commits on each side, replays scenarios with a fixed seed, validates packet on the wire (CRC32c + basic SCTP parsing), and writes artifacts (results.json</code>, packet logs, and pprof) so you can reproduce the numbers.</p>
The headline (max-burst): more throughput, less CPU, lower latency</a></h3>
This is the cleanest microbench in the suite: no loss, no delay, no jitter. Just a burst of messages in both directions. Comparing non-rack<->non-rack</strong> vs rack<->rack</strong> (There are similar improvements even when comparing rack<->non-rack</strong>):</p>
metric</th> main (baseline)</th> RACK</th> delta</th></tr></thead>

goodput</td> 234.55 Mbps</td> 316.42 Mbps</td> +34.9%</strong></td></tr>
CPU time (cpu_seconds</code>)</td> 0.0560 s</td> 0.0441 s</td> −21.3%</strong></td></tr>
goodput / CPU-second</td> 4,189</td> 7,177</td> +71.3%</strong></td></tr>
latency p50</td> 16.37 ms</td> 11.86 ms</td> −27.5%</strong></td></tr>
latency p99</td> 36.95 ms</td> 27.84 ms</td> −24.6%</strong></td></tr>
</tbody></table>
That +71% throughput-per-CPU</strong>  is simply the goodput measured (Mbps) divided by the run's cpu_seconds</code>. Non-rack cruised at ~234 Mbps using ~0.056 CPU seconds (~4,189 Mbps/CPU-s), while RACK sustained 316 Mbps with ~0.044 CPU seconds (~7,177 Mbps/CPU-s). That gap is the proof that rack delivers ~71% more work per unit of CPU.</p>
Test setup</a></h3>
To test RACK, we ran these test profiles to compare how RACK performs against main (baseline):</p>

max-burst</strong> - "how fast can we go" with no delay, loss, or reordering; it targets the raw transport path:</li>
</ul>
Goodput jumps +34.9% (234 ->316 Mbps) while CPU seconds drop by 21% (0.056 ->0.044 s) and p50/p99 both fall by ~25%. RACK now delivers ~71% more Mbps per CPU-second.</p>

handshake</strong> - same burst pattern, this time including</strong> the COOKIE/SHUTDOWN handshake, so we exercise setup timers:</li>
</ul>
Goodput climbs +15% (237 ->272 Mbps) while latency stays basically flat (15.65 ->15.99 ms for p50, 35.27 ->33.25 ms for p99), which confirms that the faster throughput comes without slower ACK paths.</p>

unordered-late-low-rtt</strong> - minor delay/jitter (10 ms) but unordered delivery to simulate packet trains with mild disorder:</li>
</ul>
Minor latency and throughput noise. Both branches still pass but RACK keeps the delivery steady despite small unordered bursts.</p>

unordered-late-high-rtt</strong> - large RTT/jitter (180 ms / 60 ms) with unordered delivery, so we can watch how the stack copes with latency spikes:</li>
</ul>
Very high latency due to the profile, but RACK keeps throughput comparable while completely avoiding any regressions.</p>

unordered-late-dynamic-rtt</strong> - fluctuating RTT (40 ms base ±180 ms jitter) with unordered delivery to mimic burst-y network dynamics:</li>
</ul>
Both branches pass with no noticeable regressions from RACK, which shows that each branch handles jitter swings fine.</p>

congestion</strong> - ordered delivery with 2% loss and modest delay/jitter to stress-tests congestion control and SACK-driven recovery:</li>
</ul>
The loss-handling path stays green and RACK doesn't use extra CPU compared to master which shows the +35% clean-case gain doesn't cost the loss profile.</p>

retransmission</strong> - ordered with 5% loss and 20 ms jitter to force fast-retransmit/TLP scenario:</li>
</ul>
The fault case still hits retries and RACK's CPU profile actually shows more JSON/packet-logging work but that's what we expect during retransmit storms.</p>

reorder-low</strong> - unordered with 1.5% loss plus deliberate reordering to exercise scheduler/queue behavior under loss＋reorder:</li>
</ul>
Goodput improves +44% (1.79 Mbps -> 2.58 Mbps) and the run finishes faster (~3.70 s vs 5.34 s), so RACK dominates the low-rate reordering scenario.</p>

burst-loss</strong> - unordered with 4% loss and 50 ms jitter to push retransmit/recovery under heavy loss bursts.</li>
fragmentation</strong> - oversized payloads require chunk fragmentation/reassembly to verify large-message handling:</li>
</ul>
Nothing improved or got worse.</p>

media-hevc</strong> - A real-world use case with video: one-way stream, paced HEVC frames (~25 fps), 3% loss, ~1200-byte max payload, across a ~13-14 Mbps link (taken from a real-world use case of sending DRM media over WebRTC datachannels) to ensure sustained media delivery works.</li>
</ul>
RACK hits 12.90 Mbps goodput in 2.14 s (100% delivery) while the main branch streaming to the RACK branch sits at 11.34 Mbps in 4.66 s. That's a 2x faster finish!</p>
We also have 3 negative tests to ensure that any corruptions or errors are still being caught:</p>

fault-checksum</strong> - corrupts every 7th DATA chunk's checksum so receivers must drop it and log the error.</li>
fault-bad-chunk-len</strong> - mangles the chunk length field every 7th chunk to validate length checks/parsing.</li>
fault-nonzero-padding</strong> - corrupts padding bytes every 7th chunk so padding validation and chunk isolation logic are exercised.</li>
</ul>
In both branches, these cases fail (as desired), which confirms that both branches detect the corruption and that there is no regression in behavior.</p>
CPU flamegraphs</a></h3>
The flamegraphs below show the CPU profiles for max-burst runs. You can see in the metadata that the RACK profile captured 20ms of samples</strong> (9.95% of 201.08ms duration) versus the master profile's 40ms of samples</strong> (19.86% of 201.45ms duration) – exactly half the sample count for a similar duration, which directly supports the efficiency claims.</p>
</p>
</p>
Why RACK behaves better (not just "goes faster")</a></h3>
RACK changes how</em> SCTP decides that something is lost and when it sends probes, so it wastes less work fixing problems that never really happened:</p>

Instead of keying almost everything off "three missing reports or an RTO fired"</strong>, RACK uses time-based loss detection</strong>: it looks at when chunks were last SACKed/ACKed and infers loss from elapsed time and the pattern of tail acknowledgments.</li>
Tail Loss Probes (TLP)</strong> send a cheap "sample" chunk at the end of a burst to flush out late ACKs. If the receiver really did get the data, it answers and the sender avoids a full retransmission storm, otherwise, the probe doubles as the retransmission you needed anyway.</li>
</ul>
In practical terms that's what the profiles and metrics are showing:</p>

We still see the same hot stack (vnet.(*chunkUDP).UserData</code>, runtime.memmove</code>, a thin layer of runtime/type helpers) in both master and RACK. It's the normal packet I/O path.</li>
With RACK, that stack is exercised fewer times per unit of useful data</strong> because there are fewer spurious retransmits and fewer "just in case" timer expirations.</li>
That's exactly how we get more goodput, lower latency, and smaller CPU profiles at the same time</strong>: RACK spends less CPU "arguing with the network" and more CPU pushing real user data through SCTP.</li>
</ul>
Spec-aligned ACK behavior and testing</a></h3>
Using SCP testing tool we were able to find some issues includes:</p>


In the initial RACK implementation, handling of transitions from high to low RTT was suboptimal due to the implementation using a global minimum for recent RTT measurements instead of a windowed minimum (the latter approach is only a "SHOULD" in RFC 8985 section 6.2.1</a>). Atsushi Watanabe quickly identified it and we resolved the issue.</p>
</li>

The earlier version of RACK implementation also handled packet reordering poorly and consumed more CPU than non-RACK. This was corrected by implementing improved active RTT measurement, following the approach described in Weinrank's work, see p. 120</a>.</p>
</li>

A minor bug was discovered (and fixed) in the initial RACK implementation where the latest RTT was not measured for every packet.</p>
</li>

We also found that Pion SCTP did not send a SACK immediately after a TSN gap, causing RACK to perform worse under moderate reordering. After fixing this behavior to align with RFC 4960 section 6.7 (surprisingly only a "SHOULD")</a>, reordering test cases showed a ~30% improvement.</p>
</li>
</ul>
Looking forward</a></h2>
Keep an eye out for even more improvements and benchmarks from our improved SCTP implementation using real-world data, as well as how we're doing it in an upcoming blog post!</p>
Thanks for reading!</p>
Credits</a></h2>
Huge thanks to the following for making this possible:</p>

Jo Turk</a> for introducing me to Pion, making SCP, answering countless questions, and so much more.</li>
Sean DuBois</a> for making Pion, finding Felix Weinrank's thesis</a>, and endless encouragement.</li>
Srayan Jana</a> for helping to bounce around many ideas.</li>
Atsushi Watanabe</a> for reviewing and catching the global minimum vs windowed minimum issue in the RACK PR.</li>
And many more people along the way!</li>
</ul>


WebRTC Survives When You Walk Out
Kevin Wang — Fri, 17 Oct 2025 00:00:00 +0000
Networking for a video call sounds simple. You send your video to the
server and server sends video back. But that mental model breaks down in
the real world. A connection that starts on stable home Wi-Fi may
suddenly need to continue over cellular as someone walks out the door.
The real challenge isn’t just connecting two points! It’s staying
connected as the network beneath you changes.</p>
With automatic renomination</a>, Pion
will now handle that completely seamlessly, allowing your media to flow
without skipping a beat. It works by periodically detecting when new
connection candidates are available and automatically switching to the
highest quality candidate, taking advantage of a feature in WebRTC called
renomination</em>.</p>
Let's dive into how that works.</p>
How does WebRTC establish connections?</a></h2>
Connections between two peers in WebRTC are established by exchanging
candidates</strong> between the two peers in a signaling phase. These candidates
can be host, server reflexive, peer reflexive, or relay candidates.
The details of the various types of candidates are better explained in
WebRTC for the Curious</a>,
but the relevant detail is that for each candidate, both peers open up
ports to communicate.</p>
</p>
Eventually, the controlling</strong> peer picks a candidate and tells the
controlled</strong> peer to use it in a Binding request. This candidate becomes
the selected candidate that media flows over. After this point, the unused
ports are closed.</p>
</p>
But wait! This handshake is pretty slow. To speed things up and get media
flowing faster, WebRTC allows media to flow over any</em> candidate before one
is formally nominated. This shaves off a few round trips between the peers,
reducing connection latency, in a process called
aggressive nomination</a>.</p>
Changing candidates with renomination</a></h2>
Now, what if we never</em> commit a candidate? The ability to send media over
any valid candidate is curious, and in fact quite useful because it means
we have multiple paths to take advantage of. This is called renomination</strong>.</p>
Under renomination, the controlling peer tells the controlled peer that it
wants to be able to renominate candidates. When the controlled peer
receives a binding request, instead of closing the ports, it leaves them
open. This allows the controlling peer to elect a new candidate at will,
which the controlled peer will accept media over, effectively enabling us
to switch candidates mid-stream.</p>
</p>
That can sound complex to negotiate but it amounts to a flag in the
connection handshakes that ensures that both parties are aware of the desire
to renominate, meaning the implementation of this feature in Pion amounts to
keeping track of a counter</a>.</p>
Now Pion is taking advantage of WebRTC's aggressive nomination state,
keeping candidates open, and has the ability to pick a new candidate.</p>
Making it automatic</a></h2>
Ideally, the WebRTC connection can do this automatically so the
user doesn't have to pick. After all, manually renominating candidates seems
useful but is a tedious user experience. Let's do it automatically!</p>
To make this work, we need to solve two problems: knowing when the current
candidate gets worse, and discovering when completely new candidates appear.</p>
Detecting when candidates get worse:</strong> Turns out, this is already built
into WebRTC. During the first connection establishment phase, candidates are
pinged, evaluated, and ranked based on round trip time, packet loss, and
type of candidate. This same ranking can be performed periodically instead
of only at initialization. So, the connection can run this on a loop and
automatically renominate candidates when the state of the network changes.</p>
Discovering new candidates:</strong> What happens when a completely new candidate
comes online? For example, if I connect to a wifi network mid-call, the
phone receives a completely new IP address.</p>
This, too, is built into WebRTC, in the form of trickle ICE! If trickle
ICE is enabled, candidate exchange can continue after the initial
connection handshake until the two sides declare that they've exhausted the
candidate search. Under automatic renomination, we go one step further and
disable the exhaustion declaration</strong>, periodically checking for new
candidates instead. When a new candidate arrives, we use the normal trickle
ICE mechanism to exchange it between the two peers.</p>
Putting it all together</a></h2>
And that ends up being enough! In summary, automatic renomination makes the
following tweaks:</p>

Signal renomination between the peers</li>
Keep all candidate ports open instead of closing them</li>
Disable trickle ICE termination</li>
Periodically ping all candidates and rerank them</li>
Renominate when a better candidate arises</li>
</ul>
With these pieces in place, Pion can seamlessly transition between different
network links. If you want to see it in action, check out
this example</a>!</p>
One really nice thing that I personally liked about this feature when
implementing it is that it felt like automatic renomination is a
clever reading of the specification: it doesn't need to reinvent a whole
new mechanism and RFC.</p>
In fact, one can go one step further and send media over all</em> candidates,
because under aggressive nomination they are all valid. This, actually, is
connection bonding for free! There are some practical considerations to
getting this working well though, but that's another blog post.</p>


Making a game with Pion
Srayan Jana — Tue, 09 Sep 2025 00:00:00 +0000
(The following was adapted from a talk I gave at DWeb Weekend 2025</a> at the Internet Archive in San Francisco on August 17, 2025)</p>
Simplifying WebRTC Datachannels for Games</a></h1>
First of all, to get some stuff out of the way, instead of using the "super complicated" WebRTC datachannels, why don't we use something simpler?</p>
Why Not Use Websockets?</a></h2>

Too slow</strong>: Most games use UDP with a reliability layer on top.</li>
Suitable for turn-based games</strong>: Fine for games that are turn-based/slow-paced, like Runescape.</li>
For more details, see:

UDP vs TCP</a></li>
Client-Server Connection</a></li>
</ul>
</li>
</ul>
The Case for Web Transport</a></h2>

Replacement for Websockets, uses QUIC instead of TCP.</li>
However, it has been in development for a long time, and I want to get started on making games now.</li>
Current issues:

Caniuse WebTransport</a></li>
Official demos on GitHub</a> do not work on Firefox on Windows.</li>
GitHub Issue</a></li>
Firefox Bugzilla</a></li>
</ul>
</li>
Potential</strong>: Will probably solve most problems once finished and would be the best choice for making a multiplayer game on the web.</li>
</ul>
Datachannels: A Hidden Gem of WebRTC</a></h2>

Feature</strong>: Lets us send unreliable packets over the web using SCTP.</li>
Advantage: We don’t need WebTransport; we can use this today.</li>
They were created in part for exactly this use case

See the reasoning for unreliable datachannels in the official specification itself</a></li>
Not to mention that one of the original implementers of WebRTC wanted to enable this usecase</a>
</li>
</ul>
</li>
</ul>
Benefits and Drawbacks of WebRTC</a></h2>
Benefits</a></h3>

Host flexibility</strong>: Do not need to host servers - players can make their own.

No more need for port forwarding or Hamachi!</li>
</ul>
</li>
Minimal server requirements</strong>: Only need one server for signaling.</li>
Community-supported specification</strong>: There are many options for making your own WebRTC-based app.</li>
</ul>
Drawbacks</a></h3>

Complex setup</strong>: Setting up WebRTC is challenging.</li>
Server dependence</strong>: You need to host or use two different servers (Signaling + STUN/TURN).

Can use Google’s STUN, but combining signaling and STUN into one server would be nice.</li>
</ul>
</li>
</ul>
WebRTC Implementations</a></h2>

libwebrtc</a> - (a fork of) the original WebRTC implementation.</li>
libdatachannel</a> - C/C++.</li>
Pion WebRTC</a> - Go.</li>
webrtc-rs</a> - Rust (using Tokio Runtime).</li>
str0m</a> - Rust (sans-io).</li>
sipsorcery</a> - C#</li>
GameNetworkingSockets</a>

Uses ICE and STUN/TURN for Peer to Peer.</li>
Created by Valve and made for Steam, which means if you've ever played a game like Counter Strike or Deadlock, you've already seen this library in action!</li>
</ul>
</li>
</ul>
Existing Game Networking Libraries Using WebRTC</a></h3>

Geckos.io</a>

A client-server abstraction for WebRTC Datachannels written in Node.js.</li>
Past experience: Really nice but a bit inefficient. See GitHub Issue</a>.</li>
</ul>
</li>
Netlib</a> - Peer-to-peer WebRTC datachannel library for TypeScript.</li>
Matchbox</a> - WebRTC datachannel library for Rust, compiles to both native and WASM.</li>
PeerJS</a> - Great for browser-only apps/games, not specifically for game networking.</li>
Netplayjs</a> - Untested but seems to work well.</li>
Godot Engine's WebRTC Native</a> - Official Godot bindings to libdatachannel for Godot’s multiplayer API.</li>
</ul>
Real-World Uses</a></h2>

WebXash3D</strong>: Reimplementation of Half-Life 1 + Counter-Strike 1.6 using WebRTC.

GitHub WebXash3D</a></li>
Xash3D Launcher</a></li>
Counter Strike 1.6 on the web</a></li>
</li>
</ul>
</li>
Hypersomnia</strong> - Open source 2D shooter made with C++ and libdatachannel

Has cross platform capabilities between both

The native Steam port</a></li>
And in the browser (On CrazyGames</a> and the official website</a>)</li>
</ul>
</li>
Github Link</a></li>
</ul>
</li>
</ul>
Why Go for Development?</a></h2>

Efficient: It just gets the job done.</li>
Pion: Has a pure-Go implementation of WebRTC.</li>
Minimal dependencies: No need for external dependencies like OpenSSL.</li>
</ul>
Why Not Rust?</a></h2>

Preference: I like Rust, but Go has a more active scene/easier help for WebRTC.</li>
Speed of development: It's faster to code in Go compared to Rust.</li>
</ul>
Why Ebitengine for Game Development?</a></h2>

Website Link</a></li>
Versatile</strong>: 2D game engine that works across various platforms.

The creator, Hajime Hoshi</a>, has ported the engine (and the Go language!) to the Nintendo Switch</a> and other game consoles</li>
</ul>
</li>
Battle Tested</strong>: Has been used for actual games like

From Madness with Love</a>

Steam Link</a></li>
</ul>
</li>
Coral & The Abyss

Steam Link</a></li>
Nintendo eShop Link</a></li>
</ul>
</li>
Rakuen

Steam Link</a></li>
Nintendo eShop Link</a></li>
</ul>
</li>
And more on the Ebitengine Showcase Page</a></li>
</ul>
</li>
Potential Crossplay</strong>: You could make a cross platform multiplayer game in pure Go that works between the Nintendo Switch, PC, and the browser!</li>
</ul>
Official Pion Example</a></h2>

We now have an official example using Ebitengine in the Pion example-webrtc-applications</a> repository</li>
This includes an bundled in signaling server so you can host your own lobby, and connect with another player.</li>
Github Link to Game</strong></a></li>
PR where it was merged in</strong>: Example WebRTC Applications</a></li>
Limitations</strong>

Right now, this can only support two players on the same computer

Could probably work between two different computers, but we would need to figure out how to setup CORS properly for the signaling server</li>
The machinery is there to support more than two players in a lobby, but as of writing this article it is currently hardcoded</a> to just two
</li>
</ul>
</li>
</ul>
</li>
</ul>
Other Games That Use WebRTC</a></h2>

Tough Love Arena</a></li>
Counterpick Labs</a></li>
2D Soccer</a></li>
</ul>
Possibilities with WebRTC</a></h2>

Hosting a Minecraft-style game without dedicated servers or port forwarding.</li>
Players host everything themselves: no need for VPN/Hamachi.</li>
</ul>

Community Acknowledgements</a></h2>

A big thanks to:

The Pion Discord</a></li>
The libdatachannel Discord</a></li>
The Ebitengine Discord</a></li>
The Rust gamedev Discord</a></li>
</ul>
</li>
And many more who made this journey possible. I stand on the shoulders of giants.</li>
</ul>


FEC with Pion
Jingyang Kang — Mon, 23 Jun 2025 00:00:00 +0000
We are excited to have FEC encoding support available in the pion/webrtc v4.1.2 + pion/interceptor v0.1.38. In this blog post, we'll explain what FEC is and how to implement it with Pion.</p>

If you already know FEC, feel free to skip to FlexFEC in Pion</a>.</p>
</blockquote>
Quick background: What is FEC?</a></h2>
Forward Error Correction (FEC) is a proactive loss-recovery mechanism:</p>
The sender transmits extra mathematically derived parity packets (e.g. xor of the media packets) along with each block of media packets. If the receiver later discovers that one or even more packets are lost, it can use the parity packets to recover the lost packets, using the XOR result of the received packets and the parity (repair) packet, all without waiting for the lost packets to be retransmitted.</p>
A simple example:</p>
Data packets</th> Parity packet</th> Recovery logic</th></tr></thead>

A</code> B</code> C</code></td> P = A ⊕ B ⊕ C</code></td> If B</code> is lost, receiver computes B = A ⊕ C ⊕ P</code></td></tr>
</tbody></table>
Because parity travels with the media, recovery happens locally and immediately. The trade‑off is more data on the link.</p>
We Already Have NACK, Why FEC?</a></h2>
Alice is sending a video stream to Bob with WebRTC. In an imaginary perfect network condition in which no packet loss happens and the network delay is 0 ms, all Alice needs to do is encode the video, pack RTP packets, and send them to Bob. What Bob needs to do is simply depacketize RTP packets and decode the payload into video frames. Everything is great except this imaginary perfect network condition doesn't exist.</p>
sequenceDiagram</span></span>
    title "Perfect Network"</span></span>
    participant Alice</span></span>
    participant Bob</span></span>
    Alice->>Bob: Video Packet n</span></span>
    Alice->>Bob: Video Packet n+1</span></span>
    Alice->>Bob: Video Packet n+2</span></span>
    Note right of Bob: Great! :)<br/>But not reality</span></span></code></pre>
Let's gradually introduce more factors into our network, to make it closer and closer to the real internet we live in. First, let's add packet loss while keep network delay still at 0 ms. With packet loss, not every packet Alice sends is guaranteed to be received by Bob. So, we need a mechanism to recover these lost packets. A straight forward approach is to make Alice resend the lost packets again. In order to notify Alice which packet is lost and needs to be retransmitted, Bob should explicitly send message to Alice, containing the sequence number of lost packets. This is NACK. With NACK in our lossy but no-delay network, all packets can be retransmitted immediately after Bob sends a NACK request. Problem solved? In this case, yes. But in the real world, no.</p>
sequenceDiagram</span></span>
    title Using NACK</span></span>
    participant Alice</span></span>
    participant Bob</span></span>
    Alice->>Bob: Video Packet n</span></span>
    Alice->>Bob: Video Packet n+1</span></span>
    Alice->>Bob: Video Packet n+2</span></span>
    alt Packet n lost</span></span>
        Bob-->>Alice: NACK n</span></span>
        Note right of Bob: Extra RTT(s) here</span></span>
        Alice->>Bob: RTX Packet n</span></span>
    end</span></span></code></pre>
Next, we are also adding delays to our imaginary network. At this point, it basically has all the characteristics of real world networks. Can our current mechanism tackle network delay + loss without hurting transportation quality? Sadly, no. If Alice packs a frame into 10 packets, Bob can't decode the frame even if only 1 packet is lost. And since NACK takes at least 1 additional RTT (more than 1 when retransmitted packets are lost too), this frame is delayed on Bob's side. To make things even worse, in modern video codecs, in order to increase compression ratio, encoders use a technology called forward prediction</strong>, which means subsequent frames relies on the data of previous frame to decode. For any frame delayed due to network loss, all subsequent dependent frames will be delayed. This can cause noticeable jank for Bob, which is bad.</p>
sequenceDiagram</span></span>
    title Using FEC</span></span>
    participant Alice</span></span>
    participant Bob</span></span>
    Alice->>Bob: Video Packet n</span></span>
    Alice->>Bob: Video Packet n+1</span></span>
    Alice->>Bob: Video Packet n+2</span></span>
    Alice->>Bob: FEC Packet</span></span>
    alt Packet n lost</span></span>
        Bob->>Bob: Recover n from n+1, n+2, and the FEC Packet</span></span>
        Note right of Bob: No extra RTT(s) here</span></span>
    end</span></span></code></pre>
FEC is a mechanism that can mitigate this. FEC stands for Forward Error Correction</strong>. Basically, it means that we can send more data to recover from potential packet losses. With FEC enabled, Alice does not only send 10 video packets to Bob, but also a few redundant packets, containing data XORed from video packets. When packets get lost during transmission, Bob can try to recover it with redundant data from FEC packets. In this case, no additional RTT is needed to recover packets lost, so video streaming is not affected.</p>
Recovery groups</a></h2>
On the sender side</a></h3>
Media packets are grouped for protection, for example:</p>

Group 1: M1, M3, M5 protected by F1</li>
Group 2: M2, M4 protected by F2</li>
</ul>
</p>
On the receiver side</a></h3>
During the transmission, some media packets might be lost:</p>
Received FEC packets are used to reconstruct missing data:</p>

F1 recover any lost packet from Group 1</li>
F2 recover any lost packet from Group 2</li>
</ul>
</p>
FEC Mechanisms in WebRTC</a></h2>
In WebRTC there are two main RTP based forward-error-correction (FEC) mechanisms. They both generate XOR parity packets, but the way they're signalled and routed is very different:</p>
Feature</th> FlexFEC (RFC 8627</a>)</strong></th> ULPFEC + RED (RFC 5109</a>)</strong></th></tr></thead>

Transport</strong></td> Travels on its own SSRC and payload-type, SDP advertises an extra repair m-section and an a=ssrc-group:FEC-FR <media-ssrc> <fec-ssrc></code> line.</td> Parity bytes are wrapped inside a RED packet that uses the same SSRC as the media. No extra m-section needed; you just add a RED and a ULPFEC payload-type.</td></tr>
Codecs</strong></td> True Codec-agnostic</strong>, works with VP8/9, H.264, AV1 … anything that produces RTP packets.</td> While codec-agnostic</strong> in theory, Chromium limits full protection to VPx and AV1. With H264 and H265 the encoder can't flag per-partition loss importance in a way that is useful for FEC. ULPFEC + Nack for codecs without Picture-ID will have to retransmit FEC packets, So ULPFEC is not used for H264 and H265.</td></tr>
Loss patterns handled</strong></td> Two-dimensional (rows × columns) parity</td> Single-direction parity</td></tr>
Implementation status</strong></td> All Chromium-based browsers can receive</em> FlexFEC; send-side</strong> support is still immature. Limited receive support in Safari and Firefox doesn't support it.</td> Send + Receive</strong> in Chrome, Firefox, Safari.</td></tr>
</tbody></table>
FlexFEC is the only option that remains fully codec-agnostic. ULPFEC does a fine job for VP8/VP9/AV1, but offers little benefit for H264 or H265.</p>
Codec-integrated FEC</a></h3>
While FlexFEC and ULPFEC live at the RTP packet layer, some codecs integrate redundancy straight into the encoded payload itself.</p>
The only such codec you'll meet in browsers today is Opus, and the way it works is when you set useinbandfec=1</code>, the Opus encoder embeds a low-bit-rate redundancy (LBRR) copy of frame N-1 inside the packet that carries frame N. If packet N-1 is lost, the decoder can reconstruct it from the tail of N.</p>
FEC isn't magic: trade-offs to keep in mind</a></h2>


Bandwidth overhead</strong>
If you add 20% FEC without raising the send cap, you'll need to reduce the video resolution or quality to stay within the cap. In good network conditions where the packet loss is minimal, FEC steals bandwidth in exchange for redundancy you don't need.</p>
</li>

Extra load on a congested link</strong>
When the loss is caused by congestion, adding more packets for FEC can deepen the problem, in congested networks, sometimes disabling FEC yields better results.</p>
</li>

Loss pattern mismatch</strong>
Random 1-2% loss is where light FEC shines. But for example, 10% bursts every few hundred ms requires high parity, which often ends up worse than just using RTX.</p>
</li>

Silent failures</strong>
Analytics may count frames as "recovered" even if the end user gets artifacts or glitches.</p>
</li>

Only packets, not outages</strong>
A 100ms network stall drops all packets, media and parity alike; so buffering or retransmission is still needed.</p>
</li>

FEC is best for high-latency networks</strong>
FEC is designed to avoid the delay of waiting for retransmissions, but for example, if your round-trip time (RTT) is under 30-50ms, do you really need it? In low-latency networks, simply retransmitting lost packets might be more efficient and better overall than sending redundant data up front.</p>
</li>
</ol>
So, start with the lowest cost protection first (Opus in-band FEC for audio, RTX for video). Add FEC only when telemetry shows sustained loss or when you have bandwidth room. FEC should be a dynamic option, not a set-and-forget checkbox that you enable for all users, and all network conditions.</p>
FEC algorithms beyond XOR</a></h2>
FlexFEC and ULPFEC both use XOR-based recovery logic to generate packets. Another family of algorithms, like Reed-Solomon</a>, can recover from more complex losses but are currently not standardized in WebRTC due to their higher computational cost and block-based design, which is not best suited for real-time use in WebRTC.</p>
If you're curious, the klauspost/reedsolomon Go library</a> offers a SIMD-accelerated implementation that's fun to experiment with.</p>
FlexFEC in Pion</a></h2>
Pion now supports basic FlexFEC encoding! You can pair it with Chromium and Safari (receive-only), and it's simple to integrate – just register the payload type in your media engine and add the interceptor, then you're good to go! More about it in Enable FlexFEC Encoding in Pion</a>.</p>
For a quick demo, you can check out the play-from-disk-fec</a> example.</p>
What to look forward to in the final FlexFEC implementation</a></h3>
If you worked with FlexFEC, you may have come across multiple draft versions, FlexFEC-03</strong> and FlexFEC-20</strong>. The final standard is published as RFC 8627</a>, and while it builds on ideas introduced in the -20 draft, it formalizes several features and adds important clarifications.</p>
Although Pion doesn't yet fully support RFC 8627, we're tracking it closely and excited about the improvements it will bring, here are some of the highlights:</p>


Flexible Masking (Selective Protection)</strong>:
RFC 8627 introduces a flexible mask</strong> mode, allowing senders to protect an arbitrary subset of source packets. Instead of a fixed pattern, it sends a bitmask indicating exactly which packets are covered. This can be used to implement smarter FEC strategies, such as protecting only keyframes or higher-priority codec configuration packets.
-> RFC 8627, Section 1.1.4</a></p>
</li>

Hybrid FEC and Retransmission Support</strong>
The final spec explicitly supports both proactive FEC</strong> and RTP retransmission</strong>. It also requires that when FlexFEC is used alongside another retransmission mechanism, the answer in an SDP negotiation must use FlexFEC only, ensuring consistent protection handling.
-> RFC 8627, Section 1.1.7</a></p>
</li>

Improved Scalability and Consistency</strong>
RFC 8627 addresses scalability limitations from earlier FEC mechanisms like RFC 5109</a> and RFC 2733</a>. It also reuses the RTP repair stream format</strong> defined in RFC 6363</a>, making it more consistent with modern RTP practices and easier to interoperate with existing tooling.
-> RFC 8627, Section 1.1.1</a></p>
</li>

Packet and Header Format Clarifications</strong>
The final spec clearly defines key header fields. For example, the R bit</strong> distinguishes retransmission (R=1) from FEC repair (R=0), while the F bit</strong> selects between flexible-mask mode (F=0) and fixed-offset mode (F=1). These explicit semantics simplify implementation logic and parsing.
-> RFC 8627, Section 6.2</a></p>
</li>
</ol>
Enable FlexFEC Encoding in Pion</a></h3>
You can use the helper for quick setup:</p>
err</span> =</span> webrtc</span>.</span>ConfigureFlexFEC03</span>(</span>fecPayloadType</span>,</span> mediaEngine</span>,</span> interceptorRegistry</span>)</span></span></code></pre>
Or setup codec and interceptor by hand:</p>

Register FlexFEC03 codec to MediaEngine</li>
</ol>
err</span> =</span> mediaEngine</span>.</span>RegisterCodec</span>(</span></span>
    webrtc</span>.</span>RTPCodecParameters</span>{</span></span>
        RTPCodecCapability</span>:</span> webrtc</span>.</span>RTPCodecCapability</span>{</span></span>
            MimeType</span>:</span>     webrtc</span>.</span>MimeTypeFlexFEC03</span>,</span></span>
            ClockRate</span>:</span>    90000</span>,</span></span>
            Channels</span>:</span>     0</span>,</span></span>
            SDPFmtpLine</span>:</span>  "</span>repair-window=10000000</span>"</span>,</span></span>
            RTCPFeedback</span>:</span> nil</span>,</span></span>
        }</span>,</span></span>
        PayloadType</span>:</span> 118</span>,</span> //</span> You can change this</span></span>
    }</span>,</span></span>
    webrtc</span>.</span>RTPCodecTypeVideo</span>,</span></span>
)</span></span></code></pre>

Register FlexFEC interceptor</li>
</ol>
fecInterceptor</span>,</span> err</span> :=</span> flexfec</span>.</span>NewFecInterceptor</span>(</span>)</span></span>
//</span> handle error...</span></span>
interceptorRegistry</span>.</span>Add</span>(</span>fecInterceptor</span>)</span></span></code></pre>
That's it! But with a few catches:</p>

The order of the FlexFEC interceptor in the chain of interceptors matters. It should be registered before TWCC header extension interceptor</a> and NACK responder interceptor</a> if you are using these.</li>
If you are using congestion controller, be aware of the bandwidth FEC is taking, you should set your encoder's bitrate to cc_estimated_bandwidth - fec_bitrate</code>. By default, for every 5 media packets, 2 FEC packets will be produced, so expect roughly 40% bitrate overhead. You can tune FEC parameters to reduce or increase it.</li>
Enabling FlexFEC encoding may introduce significant CPU/RAM overhead.</li>
Current implementation does not protect media packet batches if they contain missing or reordered packets. So, it works best with senders that packetize media themselves.</li>
</ol>
Configuring FlexFEC Encoder Interceptor</a></h3>
You can customize FEC protection by tweaking NumMediaPackets</code> and NumFECPackets</code> using the interceptor's options</a>.</p>
It will result in a protection profile which uses NumFECPackets</code> FEC packets to protect each NumMediaPackets</code> media packets. Interleaved protection will be used, which means that media packet X</code> will be protected by FEC packet (X mod NumFECPackets)</code>.</p>
Verify that FlexFEC is Enabled</a></h3>

On sender side, which is within Pion, you can log the packets using the packetdump interceptor</a>.</li>
On receiver side, for example, in Chromium, you can go to chrome://webrtc-internals</code> and search for the stats graph of fecBytesReceived</code>.</li>
</ul>
Credits</a></h2>
Huge thanks to everyone who contributed to this project: Aleksandr Alekseev</a>, Jingyang Kang</a>, Jo Turk</a>, Thomas Pouget-Abadie</a>.</p>


Pion Opus
Sean DuBois — Sun, 02 Oct 2022 00:00:00 +0000
I am so excited to announce that pion/opus</a> is now available. It is a Pure Go implementation
of the Opus Codec</a>. pion/opus is far from done though. It is far enough along for others to start playing
with and contributing too though. We still have a lot more work to do. If you are interested now is a great time to get involved!</p>
Now that the code is shared I wanted to share my goals, what I learned and hopes for the future.</p>
Why Opus?</a></h3>
I had a few a different reasons for starting this project. Some of the goals I had from the very begining, others I picked up along the way.</p>
Complete the WebRTC Stack</strong> -
Implementing Opus was a clear next step for the Pion project. For the last 5 years we have been developing different parts
of the WebRTC Stack</a>. We have completed everything but audio and
video codecs. It feels great getting closer to the goal of having the entire software stack in one memory.</p>
Empower others to learn Opus</strong> -
When implementing pion/opus I designed it so others can learn from it. silk/decoder.go</a>
includes the relevant comments from the RFC. I made sure to use the same variable names and flow as the RFC. My hope is that developers can jump into the code
and learn quickly from the unit tests and code that follows the RFC.</p>
Enable Pion users</strong> -
Pion users have been building more and more interesting things. One developer is building Go clients that make it easier
for musicians to play together. Another developer is building clients that make it easier for his customers to bridge
protocols. The last developer is building a WebRTC/Phone bridge. All of them have talked about frustrations with cross
compiling and making C libs work with Go. I am sure more developers have been frustrated and just haven't felt comfortable
enough to discuss it.</p>
More efficent use of Opus</strong> -
I hope to expose the internals of this library so users can more efficently work with Opus. Maybe instead of running a Fast Fourier transform to implement
a talking notifier developers will only do partial decode. I would also like to build a bitstream analyzer using this library to help users trying to learn Opus.
I am not sure what will happen, but it is always suprising what developers build when you give them more tools.</p>
Proving Myself</strong> -
Codecs were always a daunting topic to me. I spend a lot of time using codecs, but never really understanding them. It felt like a
chance for growth to take this on. I also get asked lots of questions about coded media and have to answer in generalizations. I
hope in the future I can give people more concrete answers after this project is complete.</p>
Inspire more Go media projects</strong> -
Go is a incredibly powerful language. You can build scalable, secure and maintainable software quickly. If you have the libraries you need. I believe
having a Opus implementation will inspire other developers to join in the challenge of making Go better for media development. I want a future with more
and more Go media projects, that can't happen unless we write these libraries.</p>
Get a new group of developers involved in Pion</strong> -
Pion has primarily been WebRTC users. I hope that by having a Opus implementation it will bring a new set of eyes on the Pion project.
I think all the existing Pion projects would benefit greatly from developers joining with different opinions and backgrounds. Personally
it will be motivating to work with new people who have new demands. The pressure of Pion has lowered lately as the core libraries have stabilized.</p>
Create opportunity for contributors</strong> -
My favorite part of working on Pion is seeing the opportunities it creates. It brings me a lot of happiness watching developers grow as they contribute to Pion.
They become better programmers and collaborators. Pion also provides lots of great career opportunities. I have see multiple contributors get high paying
jobs because of the skills developed while working on Pion. It has ranged from new developers who haven't finished college to seasoned developers with decades of experience.
Getting involved in Open Source can be rewarding for anyone.</p>
What I learned</a></h3>
Not as much about audio as I hoped! When implementing this codec I spent a lot of time reading RFC6716</a>.
The RFC talks about the how, and not the why though. It is possible to learn the high level details like LPC</a>.
I wasn't able to find any details about the particulars of SILK itself. When this project is done I hope to go research that and write it down.</p>
A few implementations of Opus exist. libopus</a>, concentus</a>, lu-zero/opus</a>
and FFMpeg</a>. If you want to learn Opus I found the Rust implementation and FFmpeg the easiest to understand.
It is surprising how most of these efforts were driven by a single person!</p>
The Opus community is very helpful. A special shout out to Luca Barbato who responded to many emails from me. Without his help I would have never made it this far.
The developers in the #opus libera channel were very helpful also. Timothy Terriberry took a lot of time answering my questions on IRC. He also provided earlier/simpler
implementations of Silk which helped me understand things a lot better.</p>
Future</a></h3>
I don't know how much actual usage this project will see. I think I could have spent my time better elsewhere, but I am happy that I didn't quit. Quite a few times I wanted to give up.
I held on to the idea that it is better to finish late and have something to show, then quit and have nothing at all.</p>
My prediction/hope is that this project will give me an opportunity to work with smart and interesting people. I anticipate that I will get some contributors that want to learn.
I also think this project will mean that webrtc-rs</a> will get a Opus implementation. With this and rav1e</a> I see
a future for Pure Rust WebRTC clients!</p>


ION, grown up and moving out.
Sean DuBois — Tue, 19 Jul 2022 00:00:00 +0000
Going forward the ION project will now be hosted at ionorg/ion</a>. ION itself is not changing in any other way.
We are just making these changes to more clearly reflect the actual leadership and organization of the project. Changes like this aren't free
though. Users may have to change imports, tooling and other software. We apologize for that, but hope these benefits are worth it.</p>
Recognize the actual leadership of ION</a></h3>
The leadership and contributors of ION and Pion are often conflated. Often I will have users ask me questions or even thank me for ION.
This confusion is demotivating for the people actually doing the hard work. I worry that long term it will hurt the project.</p>
ION was created by adwpc</a> and CloudWebRTC</a> in March of 2019. As the project
has progressed it saw significant changes from tarrencev</a> and OrlandoCo</a> who
made architectural changes and other leadership. ION also has a healthy community of developers answering questions and building tools.
TannerGabriel</a>, leewardbound</a> and billylindeman</a>
always seems to be helping users on Slack</a></p>
I hope by separating ION and Pion it will make the leadership clearer and help users reach the right people.</p>
Support all the Servers/SFUs written with Pion</a></h3>
At the time no other Pion based servers existed. Pion users were requesting an easier solution for building servers, so it made sense to make one.
As time went on developers outside the Pion project built these new and interesting WebRTC server projects (sorted alphabetically).</p>

DuckSoup</a> - Videoconferencing tool for social experiments.</li>
Galène</a> - SFU that is easy to deploy with low resource requirements designed for lectures, conferences and tutorials.</li>
Harmony</a> - Voice and text communications platform</li>
Kraken</a> - SFU designed especially for group audio conferencing.</li>
LiveKit</a> - Room based API that is horizontally-scalable out of the box.</li>
Plug-N-Meet</a> - Providers higher level WebRTC tools with plugins for Joomla, Moodle, Wordpress</li>
</ul>
With ION existing under the Pion organization users would unfairly ignore these other projects. I want to celebrate and encourage everyone who
is building things with Pion.</p>


Pion is a nonprofit project
Sean DuBois — Mon, 27 Dec 2021 00:00:00 +0000
Pion saw its first commit Wed Mar 7 2018. I didn't have grand aspirations
for the project. It felt important</a> at the time to make WebRTC more
accessible. Proprietary alternatives were being proposed and they
didn't have technical advantages, they were just easier to use.
I enjoyed solving issues one at a time and helping users
use Pion.</p>
Interest in WebRTC then exploded in 2020. Lots of new demand for WebRTC
software popped up over night. New developers started learning it,
companies were founded around it and lots of VC money started entering
the space.</p>
I was approached by a few VCs who asked if I had any plans to do something
commercial with Pion. It sounded exciting and I would have been fun to work
on it full-time. In the end I decided that it may benefit me personally, but
would be a net negative. These are the reasons I decided against it.</p>
I have seen many for profit projects not have these issues. I just don't
believe I have the skills/enviroment/team to make it work.</p>
Reduce opportunities for individuals</a></h3>
Working on Open Source early in my career had a huge impact on me, and I
hope that working on Pion can do the same for others. A few milestones that
mean a lot to me.</p>

First job in video because of a project they built with Pion.</li>
First speaking opportunity talking about a Pion contribution.</li>
Company that uses Pion hiring multiple contributors and offering much better salaries.</li>
A contributor that is an expert in RTC returning to the field through Pion</li>
Companies created by first time founders using Pion</li>
</ul>
If Pion was for profit I don't think I could create an environment where this would
all be happening. I can focus on giving individuals the best outcome and not worry about
a business.</p>
Harder to be a community</a></h3>
For me Pion isn't just software, but a community to build things with
others who are passionate. I love seeing all the projects that people
share in Slack</a>. I remember lots of late night
debugging sessions and difficult bugs overcome. I remember the excitement
when WebRTC for the Curious</a> first started.</p>
I don't think those relationships would have been made if Pion was a for profit
project. I would have to prioritize issues effecting the business. I want to create
an environment where developers can pour their passion into problems that interest them.</p>
Competition with contributors and users</a></h3>
I enjoy having an open discourse with the companies and projects that use Pion. I love
learning about what people are building and learning from them. I am filled with so
much joy seeing all the different use cases.</p>

Robotics</a> - Makes it possible to build robots that make hospitals safer.</li>
Cloud Ready WebRTC</a> - Making it so any company to build and scale WebRTC</li>
Low/No-Code WebRTC</a> - Empowering the next generation of builders so they don't need to learn the details.</li>
Application Streaming</a> - Users who can't afford the latest hardware can still access everything.</li>
Collaborative Gaming</a> - Multiplayer NES in the browser, don't let distance stop you from spending time together.</li>
P2P CDN</a> - Reduce network strain by sharing what you have already downloaded. Bringing P2P to everyone!</li>
Digital Signal Processing</a> - Real time translations. Allow users to communicate with no shared language.</li>
Computer Vision</a> - Process gestures and expressions. Make video conferencing feel more natural.</li>
</ul>
All these projects have caused Pion to grow in different ways. They all have different needs and pushed Pion in lots of good ways. If I started
building competing software I would lose out on learning all these stories. Also lots of great Open Source projects Using Pion</a>
that I have learned from.</p>
Less diverse contributions</a></h3>
If I worked on Pion full-time more issues would be addressed by me directly.
I am not sure this would be better for the project long term. Pion’s growth and
quality is because we have so many contributors who bring a wide variety of opinions
and experiences.</p>
Things like WASM Support, ORTC and Interceptors came from designs by other developers.
If I had done everything myself I don't believe it would be nearly as good.</p>
Divide between employees/community</a></h3>
All community members are equal in the Pion ecosystem. Each developer only
needs one approval to merge their PR. I don't ever want there to be a
split. Employees of 'Pion Inc' would be viewed differently and would have access
to resources others wouldn't have.</p>
It is important to me that Pion is welcoming and inclusive to all.</p>


Why WebRTC?
Sean DuBois — Sun, 05 Dec 2021 00:00:00 +0000
This is a question I get in lots of different forms. Why should I use WebRTC instead of my
preferred video protocol? Why does WebRTC need to be in my browser? Why does WebRTC have to
be so complicated? Why do you continue to work on WebRTC projects?</p>
These are my answers to the Why's of WebRTC. Lots of nuance exists, and these answers are heavily
influenced by the work I do. I have been working with WebRTC since 2013. In that time I worked on two implementations Pion</a>
and KVS WebRTC</a>. I also co-authored a book WebRTC for the Curious</a>.
I have used WebRTC at startups</a> and corporate</a> projects.</p>
WebRTC supports more then just your use case</a></h3>
WebRTC supports a diverse set of use cases. It is hard to appreciate all of them when you are focused on your problem.</p>

Multiple tracks of bi-directional audio and video</li>
Media negotiation, codecs and media events like adding and removing tracks</li>
P2P connectivity with optimal path discovery, Client/Server can be built as well.</li>
Binary and text communication, lossy and unordered data with flow control APIs</li>
Mandatory encryption, secure by default and requires no developer configuration</li>
</ul>
Supporting a wide set of features has some big benefits. Most WebRTC developers only care about a specific use case, and
improve WebRTC for their needs. However, they coincidentally improve it for everyone. Lots of money has been invested in WebRTC by
conferencing companies. These changes have empowered Open Source project, startups and others that don't have the same resources.
Projects like Snowflake</a> are only possible because others pushed WebRTC to it's present day quality.</p>
WebRTC is a standard</a></h3>
WebRTC is a IETF/W3C standard. Multiple individuals have been involved since its creation. Beyond that WebRTC is made up of multiple existing
standards. The majority of what WebRTC defines is how to combine these existing complicated subsystems.</p>
I use WebRTC because I know it has a safe future. Protocols that are owned by a single corporation or project can be changed at any
time. As an individual you also have a chance to influence WebRTC. You can get involved in the standards or write your own WebRTC software and tools.</p>
WebRTC has multiple implementations and is in the browser</a></h3>
WebRTC is available in every major browser and most major languages. WebRTC isn't just a browser technology!</p>


C#</a></p>
</li>

C++</a></p>
</li>

C++</a></p>
</li>

C++</a></p>
</li>

C</a></p>
</li>

Elixir</a></p>
</li>

GStreamer</a></p>
</li>

Go</a></p>
</li>

Node.js</a></p>
</li>

Python</a></p>
</li>

Rust</a></p>
</li>

Typescript</a></p>
</li>
</ul>
You also have a wealth of servers to choose from</p>


ant media server</a></p>
</li>

galène</a></p>
</li>

ion-sfu</a></p>
</li>

janus-gateway</a></p>
</li>

jitsi</a></p>
</li>

kurento</a></p>
</li>

licode</a></p>
</li>

livekit</a></p>
</li>

mediasoup</a></p>
</li>

medooze</a></p>
</li>

Red5 Media Server</a></p>
</li>
</ul>
WebRTC's wide availability makes it easy to build with. Developers can use the language, paradigm and tools they are most comfortable with.
Just worry about learning WebRTC and not a new language. WebRTC also has a high bus factor. If a single developer or company stops supporting
their software you have lots of alternatives.</p>
WebRTC supports all skill levels</a></h3>
You can build a simple 1:1 audio+video communication application in 50 lines of Javascript. You don't need any knowledge of VoIP or networking.
WebRTC also allows a great degree of flexibility if you have more niche needs. You can ship your own Congestion Control and Error Correction for media.
You can choose your own algorithm for path selection when establishing the connection. Even if you make these changes you will still be standards compliant
and can connect to existing WebRTC clients!</p>
WebRTC's accessibility allows a wide range of applications to be built with it. You don't need to be a video and networking expert to build your
cool idea. I continue to stay involved with WebRTC because it has a rich ecosystem of developers. Life becomes stale if you build the same thing
over and over again. With WebRTC I have had a chance to interact with so many different projects. Remote surgery, drones, robotics, file sharing,
one hundred thousand viewer events, sub-second bidding for cattle, game streaming, VPNs in the browser, real-time voice translation, and these are
just the public projects! WebRTC lets beginners bring their interesting ideas to life.</p>


DTLS 1.3 Is Coming to Go: An Implementer's Perspective
Theodor Midtlien — Thu, 01 Jan 1970 00:00:00 +0000
What Is DTLS?</a></h2>
DTLS is the Datagram (UDP) variant of TLS. You might not be familiar with TLS, but you use it extensively!</p>
TLS is the protocol used for secure communication across the Internet (and intranets). Most web browsing and many email and VPN services run over it.</p>
(D)TLS solves three things:</p>

Encryption</strong> - Prevent attackers from seeing the protected data</li>
Authentication</strong> - Confirm who you are communicating with</li>
Integrity</strong> - Detect tampered or corrupted records</li>
</ul>
To accomplish this, TLS standardized a handshake so two TLS peers can agree on a cipher suite and keying material. DTLS extends and modifies the TLS
handshake so it can work over a lossy protocol.</p>
If you are curious about deeper details on the protocol, see WebRTC for the Curious - Securing</a>.</p>

DTLS 1.3 Upgrades</a></h2>
DTLS 1.3 had quite a few changes. These are the ones we noticed and found most exciting.</em></p>
Uses Less Data</a></h3>
Each DTLS 1.2 record has a header with metadata required for the session.
Each record uses a fixed 13-byte</strong> record header:</p>
      0 1 2 3 4 5 6 7</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |  Content Type |  1 byte</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |    Version    |  2 bytes</span></span>
     +               +</span></span>
     |               |</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |     Epoch     |  2 bytes</span></span>
     +               +</span></span>
     |               |</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |               |</span></span>
     +               +</span></span>
     |               |</span></span>
     +    Sequence   +  6 bytes</span></span>
     |     Number    |</span></span>
     +               +</span></span>
     |               |</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |     Length    |  2 bytes</span></span>
     +               +</span></span>
     |               |</span></span>
     +-+-+-+-+-+-+-+-+</span></span></code></pre>
DTLS 1.3 uses a variable-length unified header that can be as small as 2 bytes</strong>:</p>
      0 1 2 3 4 5 6 7</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |0|0|1|C|S|L|E E|  1 byte</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     | Connection ID |  optional, variable length</span></span>
     |   (if any)    |</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     |  8- or 16-bit |  1 or 2 bytes</span></span>
     |Sequence Number|</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
     | 16-bit Length |  optional, 2 bytes</span></span>
     +-+-+-+-+-+-+-+-+</span></span>
</span>
     C = Connection ID present</span></span>
     S = Sequence number length</span></span>
     L = Length present</span></span>
     E = Epoch</span></span></code></pre>

With Pion's default MTU of 1200, you could save 50 MB</strong> on a 5 GiB transfer!</strong> That might not sound like a lot, but at scale, it will add up to massive savings.</p>
</blockquote>
Connects Faster</a></h3>
DTLS 1.3 connects faster for two reasons.</p>
First, DTLS 1.3 has explicit ACKs.</strong> During the handshake, if a datagram was lost in DTLS 1.2, both sides would wait and then retransmit on timeout.
With DTLS 1.3, each side can send explicit ACK</code> messages. This allows selective or early retransmission after a datagram loss.</p>
Second, DTLS 1.3 allows a key share in the ClientHello</code>.</strong> This removes an entire round trip. The server no longer waits for the ClientKeyExchange</code>!</p>

This allows the handshake to be shrunk down to only 1 RTT</strong> (from 2</strong> in DTLS 1.2).</p>
</blockquote>
Below are simplified versions of the handshakes. Note how the client only has to wait for one server flight to receive encrypted data (Application Data)
in DTLS 1.3, versus waiting for two server flights in DTLS 1.2.</p>
DTLS 1.2</a></h4>
Client                                               Server</span></span>
------                                               ------</span></span>
                                  Flight 1</span></span>
ClientHello                       --------></span></span>
</span>
                                  Flight 2</span></span>
                                  <--------          ServerHello, ServerHelloDone</span></span>
</span>
                                  Flight 3</span></span>
ClientKeyExchange,                --------></span></span>
ChangeCipherSpec, Finished</span></span>
</span>
                                  Flight 4</span></span>
                                  <--------          ChangeCipherSpec, Finished, Application Data</span></span>
</span>
                     Server data after 2 RTTs</span></span></code></pre>DTLS 1.3</a></h4>
Client                                    Server</span></span>
------                                    ------</span></span>
                           Flight 1</span></span>
ClientHello + key_share    -----></span></span>
</span>
                           Flight 2</span></span>
                           <-----         ServerHello, EncryptedExtensions, Finished, Application Data</span></span>
</span>
                         Server data after 1 RTT</span></span></code></pre>Improved Security</a></h3>
DTLS 1.3 improves security in a bunch of ways: lots of insecure and brittle stuff was removed, and a few new things were added.</p>
Renegotiation was removed.</strong> Renegotiation was a source of bugs and security issues (CVE-2009-3555</code>) and was entirely removed. Pion DTLS never implemented
renegotiation, but we are excited not to get requests for it anymore!</p>
More of the handshake is encrypted.</strong> A big improvement is that certificates are no longer exchanged in plaintext.
This was a big fingerprinting surface that is now closed.</p>
Fragile and difficult-to-implement cipher suites are removed.</strong> Only AEAD</code> cipher suites are available now.
AEAD</code> is easier to use because encryption and authentication are done in one call.</p>
Compare Pion's use of CBC</a>
to its use of AEAD</a>.</p>
With CBC</code>, we have to do authentication and encryption separately. AEAD</code> combines authentication and encryption into one operation. Another benefit is
that AES-GCM</code> is often more performant. Since CBC-and-HMAC</code> involves two operations, you end up with more copying and instructions.</p>
Forward secrecy is required for all non-pre-shared-key sessions.</strong></p>
In DTLS 1.2, you had the option to use cipher suites with RSA</code> key exchange
or ephemeral key exchange. Ephemeral key exchange means you generate a new key for each session. RSA means the server would reuse the same key across many
sessions. If an attacker got access to the private key for RSA sessions, they could decrypt every session. With ephemeral keys, the attacker needs the keys
for each individual session.</p>
A pre-shared-key session can still be configured not to use ephemeral keys. Devices that are battery-powered or constrained
might not be powerful enough to generate an ephemeral key pair for each session, so they may use this instead.</p>
Connection Identification Added</a></h3>
One of the exciting things about UDP is its connectionless nature. A user can stream media or upload a file and switch networks without interruption.</p>
This wasn't easily possible with DTLS because it doesn't provide a session identifier. Most deployments would depend on the client's IP address and port as the session identifier.</p>
DTLS 1.3 and RFC 9146, an extension to DTLS 1.2, have a fix for this, though! DTLS record headers can now contain a unique ID for the session.</p>
You can now uniquely identify your DTLS traffic. This solves a few problems for DTLS users.</p>

Identify clients</strong> as they switch between Wi-Fi and cellular networks.</li>
Load balance effectively</strong> by routing sessions to specific servers.</li>
Support long-running sessions.</strong> An IoT device can shut down and come back days later.</li>
</ul>

Implementer's Perspective</a></h2>
Finished Is a Different Story in 1.3</a></h3>
In DTLS 1.2, you have one finite state machine. After the handshake finishes, you are done.</strong> With DTLS 1.3, things get a bit more complicated! You need to handle NewSessionTicket</code> and KeyUpdate</code>. Going into this project, I didn't appreciate that complication from just reading the IETF doc.</em></p>
This required a second state machine</strong> with its own ACK</code> handling, retransmission timer, etc. See Pion's post-handshake implementation</a>.</p>
— Sean-Der</a></em></p>
The Names Are Scarier Than the Concepts</a></h3>
The TLS and DTLS specs make things feel really complex, but the concepts underneath are actually pretty simple. The spec is dense with terms like HKDF</code>, AEAD</code>, and AES</code>. Once you learn the vocabulary, it's actually quite easy to follow!</p>
— Jo Turk</a></em></p>
The Spec Isn't Linear</a></h3>
You can implement something near the end of the spec and discover that you need to refactor everything from the record layer to the ciphers. Section 6.1</a> introduces epochs, but it isn't until Section 8</a> that it becomes clear that the client can be at epoch 3 while the server is at epoch 4, and that the state needs to be explicitly directional.</p>
Our initial implementation maintained only one active epoch and record protection per direction. Supporting KeyUpdate</code> required per-epoch state and a series of refactors. See the tracking issue</a>.</p>
— Jo Turk</a></em></p>
The Importance of Interop Testing</a></h3>
While working on Pion DTLS, Jo Turk</a> created a dedicated repository for DTLS interoperability testing: dtls-interop</a>.</p>
Testing against wolfSSL she found three issues in Pion's DTLS 1.3 implementation (and fixed them)</p>

Recognize partial handshake retransmissions</a></li>
Resume and ACK split flights</a></li>
Handle duplicated Flight 4</a></li>
</ul>
It currently tests against OpenSSL, BoringSSL (Chrome), and wolfSSL. Firefox and more will be added soon!</p>

Fixing WebRTC data-channels head-of-line blocking with RFC-8260

RACK makes Pion SCTP 71% faster with 27% less latency

WebRTC Survives When You Walk Out

Making a game with Pion

Why Not Use Websockets?</a></h2> Too slow</strong>: Most games use UDP with a reliability layer on top.</li> Suitable for turn-based games</strong>: Fine for games that are turn-based/slow-paced, like Runescape.</li>

The Case for Web Transport</a></h2> Replacement for Websockets, uses QUIC instead of TCP.</li> However, it has been in development for a long time, and I want to get started on making games now.</li>

Datachannels: A Hidden Gem of WebRTC</a></h2> Feature</strong>: Lets us send unreliable packets over the web using SCTP.</li> Advantage: We don’t need WebTransport; we can use this today.</li>

Benefits and Drawbacks of WebRTC</a></h2> Benefits</a></h3> Host flexibility</strong>: Do not need to host servers - players can make their own. No more need for port forwarding or Hamachi!</li> </ul> </li> Minimal server requirements</strong>: Only need one server for signaling.</li>

Benefits</a></h3> Host flexibility</strong>: Do not need to host servers - players can make their own. No more need for port forwarding or Hamachi!</li> </ul> </li> Minimal server requirements</strong>: Only need one server for signaling.</li>

Drawbacks</a></h3> Complex setup</strong>: Setting up WebRTC is challenging.</li>

Existing Game Networking Libraries Using WebRTC</a></h3> Geckos.io</a> A client-server abstraction for WebRTC Datachannels written in Node.js.</li>

Why Go for Development?</a></h2> Efficient: It just gets the job done.</li> Pion: Has a pure-Go implementation of WebRTC.</li>

Why Not Rust?</a></h2> Preference: I like Rust, but Go has a more active scene/easier help for WebRTC.</li>

Other Games That Use WebRTC</a></h2> Tough Love Arena</a></li> Counterpick Labs</a></li> 2D Soccer</a></li> </ul> Possibilities with WebRTC</a></h2> Hosting a Minecraft-style game without dedicated servers or port forwarding.</li>

Possibilities with WebRTC</a></h2> Hosting a Minecraft-style game without dedicated servers or port forwarding.</li>

FEC with Pion

Pion Opus

ION, grown up and moving out.

Pion is a nonprofit project

Why WebRTC?

DTLS 1.3 Is Coming to Go: An Implementer's Perspective