At 3:10 on the afternoon of Monday, September 28, my paper trader posted its end-of-day wrap to Telegram. A minute later it posted the same wrap again. Then again. It kept going every minute until 4:12 CT, when I opened the database and stopped it by hand.
This was the first live-paper session across all three accounts, and it had been a quiet day. Four decisions on the primary account, all of them turned down: two TSLA breakouts and an energy pair rejected by the judgment model, and a META entry refused by risk on session_window because it landed on the last bar of the day. No orders, no fills, no positions. Nothing for a closeout job to get wrong.
Every one of those 16 duplicate posts carried correct numbers. The job did its work correctly 61 times and failed on its last step, which was writing down that it had finished.
The clue was in job_runs, not in the job
job_runs had 61 rows for closeout that night, and every one of them still said running. That is the whole bug in one table. The scheduler does not watch a job to see whether it finished; it reads the status row. A row that still says running an hour later is a job it will start again, and a row that never changes is a job it will start forever.
Two days earlier I had made the P&L calendar per-account. pnl_calendar started returning a dict keyed by (mode, account) tuples, and those stats go into the job_runs status write as JSON. A tuple is not a valid JSON key. The encode threw, the status update failed, and the one part of closeout responsible for reporting success was the part that broke.
Each restart also did the full job again, which meant re-notifying all three executors. They each re-raised the incidents left over from that morning's V-17 test, two ledger mismatches per pass, which is how one failed JSON encode turned into 32 watchdog alerts on my phone.
The guard that would not let the agent fix it
My builder agent worked out the cause while this was still running. It tried to write to the database to clear the stuck row, and its own permission guard refused the write. That was the right call: an agent reaching into production tables to patch live state is exactly what the guard exists to stop. So the loop ran on until I typed the update myself at 4:12.
What I changed, and the test that catches it next time
Commit 0bfd4a9: string keys out of pnl_calendar, run_job serializes its stats before the status write, and the closeout test now asserts that the output round-trips through JSON rather than only checking the summary is correct. Jobs redeployed at af5b284.
If you run your own scheduler, the check worth adding is that any job writing structured stats has a test which serializes them, not one that inspects them in memory. A dict with tuple keys passes every assertion you would naturally write about its contents and dies the moment it leaves the process.
The thing that saved me was the session counter. Shakedown session counting is idempotent per date, so after 60-odd reruns the primary account still read 1 of 5. Had it incremented on each pass, I would have arrived Tuesday with a finished five-session shakedown built out of a single quiet day.