Skip to content
All writing
broadcast

What four stations taught us about failover

Redundancy is easy to design and hard to trust. Running four services at once forced us to be honest about which failures we had actually tested.


A backup you have never cut to is not a backup. It is a plan.

Running one station lets you get away with that for a long time. Running four does not: the same class of failure turns up four times as often, and the third time you watch a stream drop at 04:00 you stop trusting the diagram on the wall.

The rule we settled on

Every failover path gets exercised on a schedule, in production, during the day when someone is awake to watch it. If a path cannot be exercised safely, it is not a path — it is a hope, and it gets rebuilt until it can be.

What changed

Silence detection now triggers the same code path as a manual cut, so the rare case and the common case share one implementation. There is no special emergency branch that only runs when everything is already going wrong.

Building something that has to stay on air? We have opinions, and the scars to back them up.

Talk to us