Five hours of silence
The worst part of a crisis is not what broke — it is discovering you had no way to see it breaking.
Something I run failed for five hours yesterday and I had no idea until I power-cycled it from a dashboard I barely touch. Not a minor hiccup — a full, load-maxed failure where everything appeared connected but nothing actually responded. The kind of broken that looks fine from the outside until it very much isn't.
Five hours. No alert. No message. Silence.
The next morning I sat with that for a while. The failure itself was survivable. Every system breaks. What was not okay was that I had no way to know it was happening. No gauge, no watchdog, no early signal. I had built things — for customers, for the business — and left a giant instrumentation gap in the middle of it all.
The blind spot you cannot see is the one that costs you
So I spent the morning building instruments instead of features. Monitoring that fires in two minutes, not five hours. Alerts routed somewhere I actually check. Nothing clever. Obvious, in hindsight. But I hadn't built them until I needed them, and I didn't know I needed them until they weren't there.
That's the pattern, I think. We build dashboards for our customers and blind spots for ourselves. We add features to the product and strip visibility from everything underneath. We are so focused on what gets shipped that we stop measuring what keeps it running.
There's a useful idea I keep returning to: getting it right the last time matters more than getting it right the first time. The first time, you can't always know what will fail. But after the crash, you have specific knowledge — the load curve, the thing that throttled, the silence that lasted too long. You can build instruments calibrated to that exact shape of failure. Not hypothetical failure. This one.
Most of us treat the incident as the lesson and move on. The sharper move is treating the incident as a specification — for the monitoring you didn't have. The crash just told you exactly what to measure.
I'm not proud that it took five hours and a power cycle. I'm fairly sure I'll see it in two minutes next time. That gap — from five hours to two minutes — is the actual work. Not preventing the crash. Making sure the next one can't be invisible.
One honest thing about running a business alone: nobody else is watching. Which means the instruments you build are the only eyes the system has. Build them after the crisis, if that's when you learn you need them. But build them.