Pion Interceptor: WebRTC for Hackers

Sean DuBois —

Pion was started with unique use cases in mind. Existing WebRTC software was very good at handling popular use cases. It was difficult to build unique/niche things with it though. This idea has always been at the core of Pion. Pion Interceptor was designed with the idea that we make it easy for developers to build their cool ideas. Without the frustration of forking/modifying Pion.

Prior Art

Pion's Interceptor design was heavily influenced by GStreamer Pipelines. With GStreamer pipelines you have a pipeline of elements that modify media. Things like resizing, croping or rotating media. A simple visualization like this.

Video source --> Resize --> Crop --> Rotate --> Video output

We wanted the same for Pion. Provide a library of elements for users to build pipelines. The default interceptors would give a good experience. Or you can bring your own to have custom behavior if you need it

Interceptor API

An Interceptor is started when Bind is called on it. Bind passes a StreamInfo that contains all the metadata about the stream. When a Interceptor is no longer Unbind is called. Then each RTP/RTCP packet is passed through in the Reader/Writer methods. This allows a user to process incoming/outgoing packets.

The following is the full API as generated by go doc

type Interceptor interface {
	// BindRTCPReader lets you modify any incoming RTCP packets. It is called once per sender/receiver, however this might
	// change in the future. The returned method will be called once per packet batch.
	BindRTCPReader(reader RTCPReader) RTCPReader

	// BindRTCPWriter lets you modify any outgoing RTCP packets. It is called once per PeerConnection. The returned method
	// will be called once per packet batch.
	BindRTCPWriter(writer RTCPWriter) RTCPWriter

	// BindLocalStream lets you modify any outgoing RTP packets. It is called once for per LocalStream. The returned method
	// will be called once per rtp packet.
	BindLocalStream(info *StreamInfo, writer RTPWriter) RTPWriter

	// UnbindLocalStream is called when the Stream is removed. It can be used to clean up any data related to that track.
	UnbindLocalStream(info *StreamInfo)

	// BindRemoteStream lets you modify any incoming RTP packets.
	// It is called once for per RemoteStream. The returned method
	// will be called once per rtp packet.
	BindRemoteStream(info *StreamInfo, reader RTPReader) RTPReader

	// UnbindRemoteStream is called when the Stream is removed. It can be used to clean up any data related to that track.
	UnbindRemoteStream(info *StreamInfo)

	io.Closer
}

NACK Example

A NACK is how WebRTC handles packet loss. A receiver sends a NACK so that the sender knows to re-send a packet. The following is a minimal example to show how it could be implemented.

type nackGenerator struct {
	interceptor.NoOp
	writer interceptor.RTCPWriter
}

func (g *nackGenerator) BindRTCPWriter(writer interceptor.RTCPWriter) interceptor.RTCPWriter {
	g.writer = writer
	return writer
}

func (g *nackGenerator) BindRemoteStream(info *interceptor.StreamInfo, reader interceptor.RTPReader) interceptor.RTPReader {
	var last uint16
	seen := false

	// Read incoming RTP Packets
	return interceptor.RTPReaderFunc(func(b []byte, a interceptor.Attributes) (int, interceptor.Attributes, error) {
		n, a, err := reader.Read(b, a)
		if err != nil {
			return n, a, err
		}
		header, err := a.GetRTPHeader(b[:n])
		if err != nil {
			return n, a, err
		}

		// Check if we have a gap in packets seen
		if seen && int16(header.SequenceNumber-last) > 1 {

			// Send a NACK back to sender
			_, err = g.writer.Write([]rtcp.Packet{&rtcp.TransportLayerNack{
				MediaSSRC: info.SSRC,
				Nacks: []rtcp.NackPair{
					{PacketID: last + 1},
				},
			}}, nil)
		}
		last, seen = header.SequenceNumber, true
		return n, a, err
	})
}

This simple NACK implementation detects gaps between packets and sends NACKs back to senders to fix a stream with packet loss.

Pre-Made Interceptors

Today we provide these pre-made interceptors:

Foot Guns

When building your interceptors keep these in mind.

  • Order Matters - Register the stats interceptor before the NACK responder so it can count retransmitted RTP packets. Registering stats after the responder means retransmissions bypass it. Incoming NACKs are counted in either order.

  • Don't forget to read - Interceptors are fired by you calling Read. If your application doesn't do a Read things like Stats or NACKs won't happen.