The cost of quiet failures
The crashes that bring your site down get fixed the same day. The ones that quietly lose your customers' work run for months.
Yesterday I fixed something that had been quietly wrong for months. Not a crash. Not an error page. Just a system that was silently dropping a small number of records - two items out of several hundred per batch - and every output still looked complete, because who counts?
That is the failure that should worry you more than outages.
An outage is loud. It pages you. Customers email. You drop everything. It gets fixed the same day, and then it is over. The quiet failures are different. They run until someone decides to count carefully, or until a customer asks why their thing looks slightly off and you trace it back six months and realize it has been wrong the whole time.
I found another one the same day: a file flagged as almost entirely corrupted. Ninety percent unreadable. The obvious move is to write it off. But the actual failure was a single byte out of alignment at the very start. Everything after it was fine. Most of the original intact, sitting behind one broken byte, labeled as garbage and about to be deleted.
Both had the same shape: quiet, stable, invisible until someone decided to measure.
Shipping is the beginning, not the end
Steve Jobs said real artists ship, and I believe it. Perfectionism is usually fear wearing a reasonable face, and waiting until something is perfect is how you never learn whether it works at all. But shipping is the beginning, not the end. A shipped thing that quietly fails is worse than a thing you have not shipped yet, because now it is out there compounding - dropping records, misreading files, building the wrong picture of how your product works, while you believe the silence means everything is fine.
The discipline I have had to build - and I am still building it - is to go looking for quiet failures. Not because I expect the worst, but because in any system you build fast and ship, some percentage of failures will not announce themselves. The ones that do, you catch fast. The ones that do not, you have to find by counting, by measuring, by asking the question you keep postponing.
Is this working, or does it just look like it is working? Those are different questions, and the gap between them is where your reputation lives.
The honest answer, most of the time, is both. Some things work exactly right. And some things have been dropping two records out of four hundred for months, and you had no idea.