Shadow Mode: Proving New Code Before You Trust It

One week. That's how long my new payment confirmation code has been running in production on Joytop — and not a single user noticed, because it wasn't deciding anything. It was only watching.

Tomorrow, September 14, 2026, that code switches to "deciding." One environment variable: PAYMENTS_CONFIRM_MODE, from shadow to enforce.

Why this much caution for one service

Joytop runs three payment providers — Payme, Click, and Alfabank/E-Comm. Each comes with its own webhook format, its own failure modes, its own retry behavior. A single PaymentProcessorService sits above all three — top-ups, subscriptions, ads, eSIM, everything passes through it.

I rewrote the confirmation part of that service: one ConfirmPaymentService, with a PAYMENTS_AMOUNT_CHECK flag that verifies the amount. The old code works, and it works where money actually moves. The new one is better — but there's a real distance between "I think it's better" and "I know it's better," and you can't close that distance with production's money.

What shadow mode actually does

The idea is simple: the new code runs alongside the old code, but it doesn't get to decide anything.

Every webhook triggers both paths. The old PaymentProcessorService — as always — confirms the payment, writes the state, answers the user. The new ConfirmPaymentService runs in parallel, works out what it would have decided — confirm, reject, what amount it sees — and writes all of it to a payment_events table. Nobody reads the result. Nobody is waiting on it.

That gets me two answers at once: does the new code crash when it runs (visible in the logs), and does it reach the same conclusion as the old code when it doesn't (visible by comparing two rows in payment_events).

Watching beats hoping

This is where most projects get it wrong: they turn shadow mode on, decide "give it a week, then flip," and forget about it. If there's a divergence, nobody sees it, because nothing is set up to show it.

On September 7 I added a cron for exactly this: every few minutes it compares the old and new results in payment_events, and pages Telegram on any mismatch. Not one message came in over the week. That doesn't mean "everything is fine" — it means "no divergence found yet." Those are not the same claim, and it's worth remembering that they aren't.

Meaning

Shadow mode isn't for speed — it's slower. A week of waiting, a cron to write, two paths running in parallel — all of it is extra work. I could have swapped the code directly and called it done once the tests passed.

Meaning: believing in new code and having it tested against real money are two different things. Shadow mode is the only way I know to close that gap without paying for the difference.

I flip the switch tomorrow. A week of a silent cron either finally has something to say — or it doesn't. Either answer is useful.

Last time you shipped something into production, how did you close the gap between believing it works and knowing it works?