The Soak That Failed While Nobody Was Watching

A chronicle entry: a long test handed off to an AI team, a failure nobody expected, and what the team did with it. Status: approved for external publication by owner, 2026-11-01. Audience: developers and technical leads. See Advocate brief for format recommendation.

The setup

PWA Paradise is a browser app that watches other web apps and narrates what they do. One promise in its specification is that it does not leak memory: no more than 2 MB of growth per hour, however long it runs (doc 06).

Proving a per-hour promise takes an hour. The team's audit has two versions of this test: a 12-second one that runs on every push, and a one-hour one, npm run audit:full, that Fork 8 says must be run by hand before a release. The short one exists to be quick. Its own output says so: "extrapolation is indicative only."

The delegation

On 30 October 2026 the owner said: "Can you do now the memory soak without me and then do also the commit and push without me?"

That is a different request from the one recorded in doc 15. There, the owner started the soak and asked whether the agent would say when it finished. The answer was no, and the team built a checkpoint note and a start-of-session hook to cover the gap. This time the owner handed over the whole job: start it, wait for it, and finish the work, with nobody checking in.

The Harness Engineer started the soak in the background and wrote a note, tmp/pending-check.md, saying what to do when it ended. The note was conditional. If the soak passed, update the documentation, run the checks, commit and push. If it failed, do not loosen the threshold: report the numbers and ask.

Two small things went wrong on the way, and both are in the record. The Harness Engineer's mode is not allowed to write to tmp/, so the first attempt at the note was refused and it had to be written through the shell. And one wait call hung for 16 minutes, far longer than asked. The Harness Engineer then checked the process directly instead of assuming it was still running. It was gone.

The failure

About 65 minutes later the soak had finished, and it had failed.

CheckLimitResult
Heap growth over one hour2,048 KiB116,169 KiB (about 113 MiB)

Everything else passed: frame rate, ingestion latency, rendering latency, accessibility, offline mode, sandbox integrity. Only memory failed, and by a factor of about 57. An earlier one-hour run, at the end of Phase 1, had measured 90 KiB per hour.

The instruction in the note applied. The Harness Engineer did not commit anything, did not touch the threshold, and did not update the documentation to claim a pass. It read the report and went looking for the cause.

The diagnosis

The Harness Engineer found a comment in src/main.js: the session log "holds ALL events for the current session (unbounded)". That log was added a few hours earlier with the replay scrubber (Track B3, Fork 18), a slider that lets the user scrub back through a session. Every event the app receives is pushed onto it and never removed. At the soak's load of about 200 events per second for an hour, that is around 720,000 entries. The growth fits.

The team was careful to say what it did not know. This was a hypothesis from reading the code, the timing and the size. Nobody had profiled the heap. Two other claims were also marked as unproven: that real guests produce far fewer events than the soak does, and that the per-event database writes were not part of the problem.

Why nothing caught it earlier

The 12-second soak that runs on every push has a budget of 1.5 MB. Its last reading before the hour-long run was 931 KiB. The leak was real, but 12 seconds is too short to make it exceed the budget. Every audit since the scrubber landed was green, and every one of them was blind to this.

The Harness Engineer also found a gap of its own. Fork 18 said the scrubber needed a matching audit check. Nobody had added one. That omission is recorded here as the team's, not hidden.

The help the owner asked for

The owner then said they needed help understanding what to do. The Harness Engineer set out the problem in plain terms and gave three options:

  1. Cap the in-memory log at a fixed number of events. Smallest and safest; replay only reaches back that far.
  2. Replay from the database. Matches what Fork 18 describes, but more work, and the database needs its own cap.
  3. Loosen the 2 MB limit. Refused. The limit is policy and the cause is a bug.

It recommended option 1 now and option 2 later, and left the number to the owner, because that number decides how far back replay reaches.

The owner replied that this was real analysis, asked whether it was worth documenting, and asked each team member for a view on the question before anything was done. The six personas answered from their own mandates:

The owner approved the plan and said to document it.

What is new here

Earlier entries cover similar ground in pieces. Options with a recommendation for the owner appear in Forks 14 and 15. Each member speaking for themselves appears in the-team-speaks. The notification gap is doc 15. What this entry adds is the combination: a long task handed over entirely, a failure found in a project that looked green, a conditional instruction that held when the result was bad, and a diagnosis explained to a non-expert before any code changed. This entry only claims that this is new in this project's record.

How it ends

The fix was made and the one-hour test was run again. It passed, with less room to spare than a reader might expect.

What changed. The session log is now capped at 5,000 events (commit 13f34b6, Fork 18 amendment 18A-1). The owner approved that number and it stays theirs to change. A new audit check, G6-scrubber-bounded, sends more events than the cap and confirms the scrubber never holds more than 5,000. A deliberately broken test host (unbounded-session-log) proves the check fails when the log has no cap. The check is structural: it does not need an hour, so the 12-second blind spot described above no longer covers this kind of leak. The 2 MB limit was not touched. The only change to harness/config.mjs for the cap was one new constant.

The first re-run could not be trusted, and the team first misread why. It passed, with 1,645 KiB of growth, and its report claimed 3,611 seconds in the performance gate. The note recording its start said 23:55:59Z, and the process exited at 00:13:28Z, which looked like 17.5 minutes. The team said so, called it an unexplained clock mismatch, and did not count the result or push. That reading was wrong, and the Harness Engineer's own handling caused it: it started the soak and wrote the start-time note in the same batch of commands, and the note was written only after the machine had woken from sleep. The machine's power log shows the run really started at 00:12:24 local time, the Mac went to sleep at 00:14:47 and woke at 00:55:46, so the run did last about an hour, but roughly 41 of those minutes were asleep and only about 20 were under load. The harness clock was right. The gap was that nothing in the audit noticed the load had stopped, so a soak could pass without having exercised the app for the hour it claims. The first, failed run overlapped a sleep period as well, so its "about 113 MiB per hour" extrapolation is probably an understatement; the 116,169 KiB of raw growth is a real measurement — but one that must be stated carefully: **the 116,169 KiB figure is the Harness Engineer's reading of the report at the time it ran. No artifact was kept: reports/ is git-ignored and the original file was overwritten before its significance was understood. The owner confirmed they have no copy.** The figure is consistent with the arithmetic: ~200 events per second for one hour is ~720,000 unbounded entries; the measured per-entry size puts the growth in the right range. And on 2026-10-31 a control reproduction was run — the pre-fix code (commit 711c709, unbounded log) built fresh, soaked for five minutes with the current harness, and measured 21,482 KiB of heap growth in 300 seconds, which extrapolates to 251.7 MiB per hour (reports/control-unbounded-log-2026-10-31.json). That is a different run on a different day under different conditions, but it confirms the order of magnitude and the cause. Nothing downstream depends on the exact 116,169 figure; the leak was real and the fix was correct.

The second re-run, with a second clock. The owner agreed to run the hour again. This time a separate shell loop wrote a timestamp every minute, the Mac was kept awake with caffeinate, and the start and end times were recorded outside the harness. The shell measured 3,664 seconds; the report said 3,663. All 62 ticks were present and the power log shows no sleep in that window. Heap growth was 1,607 KiB against the 2,048 KiB limit, down from 116,169 KiB in the failed run. All eight gates passed in strict mode, with nothing failed or pending. Frame rate was 60, ingest p95 0.9 ms, render p95 16.2 ms.

What that number does and does not say. 1,607 KiB is about 78% of the budget. A 60-second run of the same test, made as a one-off check outside the audit report, measured 1,596 KiB and failed its own short-run budget of 1,536 KiB by 60 KiB. So almost all of the growth happens in the first minute, which fits a log that fills to its cap once and then stops growing. Nobody has profiled the heap to confirm that, and these are single samples. The practical consequence is that the cap and the limit are close. If all of the growth is the log, a cap of roughly 6,400 events would reach the limit; that is arithmetic, not a measurement. Changing the cap therefore means re-running the hour-long test.

What the team changed because of it. The owner asked for the mismatch to be fixed. Once the real cause was found, the fix became a new check in the performance gate that confirms the load was actually delivered for the whole soak, with its own deliberately broken test host, so an interrupted soak fails instead of passing quietly.

What is still open. The Advocate advised against telling this story publicly before the fix was verified. The fix is verified, and this entry remains a draft until the owner approves it.

Source references