The Production Bug That Took Three Days and One Line to Fix

The short answer: what turned a three-day, intermittent, load-only production bug into a one-line fix wasn't a clever patch — it was method: refuse to touch the code until you can explain the mechanism, build a load harness to make the intermittent failure reproducible on demand, and only then let logging point you at the root cause (here, a per-request cache that was actually module-scoped and shared across concurrent requests).

Every engineer has one of these: the bug that eats days, defies every reasonable theory, and then collapses to a one-line fix so small it feels like an insult. Mine was a caching race condition that only showed up in production, only under load, and only sometimes — the worst combination of qualifiers there is. The three days it took weren't wasted. They were the method, and the method is the part worth writing down.

The symptom

Users occasionally saw someone else's account data on first page load. Rarely. Never reproducibly. The report would come in, we'd try the exact steps, and everything was fine. Support closed tickets as "could not reproduce" more than once before the pattern was undeniable: it was real, it was intermittent, and it was a data-leak-shaped nightmare. An intermittent bug that leaks another user's data is the kind of thing that gets you out of bed.

Resisting the urge to guess

My instinct — everyone's instinct — was to start changing code. Add a defensive check here, clear a cache there, ship it, and hope. I've learned the hard way that patching a bug you don't understand just moves it or hides it until it comes back worse. So the first rule I held to: do not touch the code until you can explain the mechanism. A fix you can't explain isn't a fix; it's a coincidence you're hoping holds.

The second rule: make it reproducible before anything else. An intermittent bug you can't trigger on demand is unfixable, because you can never prove you fixed it. I stopped chasing the symptom in production and spent a full day building a load harness that hammered the endpoint with concurrent requests for different users. On the real thing it happened once a day; under the harness it happened within seconds. That was the turning point — the moment it went from "mystery" to "problem."

Finding the mechanism

With a reliable repro, logging finally paid off. The leak correlated perfectly with concurrent requests landing on the same server process within milliseconds of each other. That pointed straight at shared mutable state, and there it was: a per-request cache that wasn't per-request at all.

Request A called setCurrentUser(alice), then yielded on an await. Request B, handled on the same process, called setCurrentUser(bob) during that gap. Request A resumed, called getCurrentUser(), and got Bob. The cache was declared at module scope, so it was one object for the entire process — and under concurrency, two requests were trampling the same slot. It worked flawlessly in development because development never had two requests interleaving at the right microsecond.

The one line

The fix was to stop sharing state across requests — scope it to the request itself instead of the module.

Swapping a module-scoped object for AsyncLocalStorage was effectively one meaningful line of intent. The harness went green and stayed green.

What the three days actually bought

The lesson isn't "watch out for shared mutable state," though you should. It's the order of operations. The days went into reproducing and understanding; the fix itself took minutes. Had I started patching on day one, I'd have shipped three plausible non-fixes and the leak would still be out there, rarer and scarier. Reproduce before you theorize, understand the mechanism before you touch a line, and let the fix be the small thing it usually is once you actually know what's wrong. The one-line fix isn't the achievement. Earning the right to write that one line is.