Deploying a new version of a real-time media server used to have a direct user cost: Kubernetes sent SIGTERM, the SFU exited, and everyone connected to it was interrupted. We reduced the blast radius by deploying off-peak, but scheduling was a workaround rather than a design.
The system we were changing
An SFU, or Selective Forwarding Unit, receives media from participants and forwards it to everyone who needs it. Each player keeps a websocket connection to one video router and connections to the SFUs involved in their current conversation. Routers assign publishing SFUs and synchronize SFU metadata through Redis.
The important distributed-systems wrinkle is that a player publishing to an SFU may be connected to a different router than another player using that same SFU. Router state is synchronized eventually, not atomically.
Requirements and constraints
- A routine deployment should not immediately interrupt an ongoing conversation.
- Engineers must be able to bypass the grace period when a bad server needs to disappear immediately.
- The first version should fit the existing player, router, SFU, and Redis architecture rather than replatforming it.
- Most conversations were expected to finish within a 60-minute maximum drain window.
The state model
We added an application-level SFU status to the metadata routers already synchronized. “Cordoned” here does not mean Kubernetes cordoned the pod; it means the application should stop receiving new work while existing conversations finish.
type SfuStatus = "ONLINE" | "CORDONED" | "TERMINATING";
type SfuState = {
address: string;
load: number;
status?: SfuStatus; // missing means ONLINE during rollout
};The drain sequence
- 1Kubernetes
Sends SIGTERM and starts a replacement SFU.
- 2Cordoned SFU
Marks itself cordoned and emits cordoned-sfu.
- 3Router A
Updates local state and immediately notifies its players.
- 4Router A → Redis
Pushes the SFU status on the next sync interval.
- 5Redis → Router B
Eventually propagates the status to other routers.
- 6Router B
Notifies its players that the SFU is cordoned.
- 7Client
Checks whether moving would interrupt a conversation.
- 8Client → Router
Requests a new SFU address, then reconnects.
When the client receives cordoned-sfu, it decides whether moving is safe. For the first iteration, “safe” reused an existing heuristic: the player was at their desk and not in an ambient meeting. If safe, the client requested reassignment and switched to the returned SFU address. Otherwise, it stayed until the conversation ended.
Why the client makes the decision
Rejected: router decides
The router lacks conversation context
Routers understand SFU availability, but did not know enough about whether a player was actively consuming conversational media. Adding that knowledge would have widened the project considerably.
Rejected: SFU broadcasts
Direct notification creates a race
A client could hear that an SFU was cordoned before its router had synchronized the update, ask for reassignment, and be sent back to the retiring SFU.
Chosen: router propagates
State first, decision second
Routers propagated authoritative availability while clients used their richer local context to decide when to move.
Keeping the process alive, but not forever
Kubernetes’ terminationGracePeriodSeconds became the maximum drain duration. An empty SFU could exit early. Shortly before the maximum elapsed, an SFU could run its final termination sequence so Kubernetes would not kill it mid-cleanup.
We also needed an emergency path. A manually triggered deployment workflow wrote a forced-termination timestamp to Redis. Routers noticed the change, sent EXIT to connected cordoned SFUs, and those SFUs immediately ran their termination routine.
Rolling out a schema change safely
The Redis schema had to support mixed versions. Old routers needed to ignore the new status. New routers needed to interpret a missing status from old SFUs as online while load messages were still arriving. We introduced the optional field before activating behavior so data could converge across deployments.
What we deliberately did not choose
Kubernetes finalizers could have kept an SFU alive until every conversation ended. But an SFU removing its own finalizer would require broader pod write access, while a custom controller would require new metrics, deployment machinery, and failure handling. The bounded grace period had known shortcomings and fewer moving pieces, so we shipped it with measurements.
Costs, risks, and measurements
- Overlapping old and new fleets temporarily increased compute spend, especially during back-to-back rollouts.
- Multiple SFU versions could coexist for up to the drain window, raising the importance of backwards compatibility.
- A compromised server could remain alive longer, making the forced exit path a security requirement rather than a convenience.
- We tracked time to termination, producers and consumers at exit, forced evictions, and conversation duration after a drain began.
This followed the approach I described earlier in production observability as continuous testing: pre-production tests verify the protocol we designed, while release cohorts show whether it improves real conversations.
Where cascading changes the problem
This flow assumes an SFU’s active work is represented by its connected players. With cross-SFU forwarding, a server can also be an internal node carrying a producer to downstream routers. Draining that node will eventually require rebuilding those pipe dependencies before exit.
I explore that next layer in scaling MediaSoup across SFUs with PipeTransports.
The design did more than make deployments gentler. Safe rotation was a prerequisite for autoscaling: a system cannot confidently remove underused capacity if doing so interrupts customers. Graceful draining turned server removal from an outage-shaped event into a lifecycle the product could understand.