Graceful SFU draining began with a user-facing goal: rotating video infrastructure should not abruptly end active conversations. The first version achieved that by letting busy servers remain alive while clients moved away at safer moments.
After shipping, the design met constraints that were difficult to see from inside the video stack. A longer termination window helped calls, but it also changed the behavior of cluster maintenance and upgrade workflows. One external timeout was shorter than the grace period we had designed around.
The lesson was not that the original approach was wrong. It was that infrastructure motivations need to be written as explicit requirements. I had described deployments and autoscaling in detail, but I had not called out cluster upgrades as a first-class use case. Because it was absent from the document, I did not test it in a development cluster.
This changed how I run design reviews. A review should not only ask whether the proposed sequence works. It should ask who else shares the lifecycle being changed, which operational workflows rely on the old behavior, and what hidden clocks or quotas exist outside the application.
The launch also reopened a strategic question: should we invest more in avoiding server removal, or make client-side switching resilient enough that removal is routine? Graceful draining reduced immediate disruption, but a truly robust client could turn server rotation into a normal event rather than an exceptional one.
Post-launch investigations are part of the design, not an epilogue to it. Communication cadence matters, especially when a change touches real-time media. A technically narrow incident can involve client behavior, distributed state, deployment tooling, and human coordination all at once.
The best infrastructure projects create options. Draining made safer deployments possible and cleared a path toward autoscaling. The surprises after launch showed where the next investments should go.