The Golden-Change Playbook

Who this is for — anyone maintaining a golden/snapshot corpus, especially one an agent can regenerate.

Read this if — you are about to run a test command with -update.
Skip this if — you have no golden files; the discipline still transfers, the commands do not.

Takeaways

1. Predict which goldens will change before you edit code; a mismatch means your model of the bug is wrong.
2. Prove the root cause with numbers and a formula, never with a screenshot.
3. -update is a ratification ceremony, not a way to make a suite green.

Visual golden files (like PNGs or ANSI snapshots) serve as the ground truth for what renderers produce. However, careless golden updates can easily ratify and hide bugs. A golden test suite is only as trustworthy as the discipline behind its last -update — one lazy regeneration and your safety net is quietly certifying the crash it was built to catch.

The following playbook, established during the development of cati’s terminal graphics engines, enforces the discipline required to fix visual regressions safely.

graph LR
    R[1-2. Reproduce & prove root cause] --> P[3. Predict which goldens change]
    P --> E[4. Fix the root, delete band-aids]
    E --> C{5. Diff matches prediction?}
    C -->|no, model incomplete| R
    C -->|yes| U[6. Update goldens deliberately]
    U --> L[7. Commit tests + code + docs together]

Important

The Golden Rule: Prove the bug numerically before you touch a line of code, and change a golden image only once you can name the specific incorrect pixel/character and explain why it was wrong.

The Diagnostics & Update Loop

1. Reproduce & Isolate  → Find the smallest input; find a control that does NOT fail.
2. Prove Root Cause     → Use numbers, not screenshots. Explain every good and bad case.
3. Predict Golden Diffs → Write down exactly which goldens will change before editing code.
4. Fix the Root         → Fix the source layer, and delete downstream band-aids.
5. Confirm Invariants   → Verify live output, add property tests, check diff count.
6. Update Goldens       → Regenerate, revert unintended edits, inspect pixel values.
7. Close the Loop       → Commit tests, code, docs, and issue references together.

1. Reproduce and Isolate

• Shrink the Repro: Reduce inputs to the absolute minimum (e.g., target width -w 6 instead of -w 80) so the raw ANSI characters can be inspected directly via cat -v.
• Find a Control: Find a parameter or size that does not reproduce the bug. If -w 6 is broken but -w 5 is clean, the difference between these two cases is your best clue.

2. Prove Root Cause (Numbers First)

• Avoid Screenshot-Driven Debugging: Visual checks confirm that something is wrong; they do not explain why.
• Write a Diagnostic Script: Create a small diagnostic script (e.g. scripts/debug_probe.go) that imports the production functions, feeds the repro inputs, and prints the coordinates and math. Delete this script once done.
• Formulate the Proof: You must be able to state the bug as a formula that holds true for every failing case and false for every passing case.

🧑‍✈️ Human holds the wheel — accepting the proof. An agent will happily produce a formula that fits the three cases in front of it; deciding that the explanation is complete is a judgement about what has not been tried yet.

3. Predict Golden Impact

• Compute the Diff Set: Before updating code, check your golden corpus. List which files will change under your formula.
• The Red Flag: If your code change later alters more or fewer goldens than you predicted, stop immediately. Your model of the root cause is incomplete. Revert your code changes and return to Step 2.

4. Fix the Root, Delete the Band-Aids

• Fix the Origin: Edit the code at the layer where the wrong calculation originates.
• Clean the Codebase: If earlier developers added symptom-suppression hacks downstream, remove them. Leaving downstream band-aids leads to codebase rot.

5. Confirm Invariants

• Live Check: Run the repro and verify it matches the expected output.
• Codify Invariants: Write unit tests that assert the underlying mathematical property (e.g., “modes must agree on vertical cell snap boundaries”), rather than just comparing strings.
• Golden Match: Run the test suite without the update flag. Ensure the failed goldens list is identical to your step 3 prediction.

6. Update Goldens Safely

• Regenerate files using the update flag (e.g., go test ./cmd/ -update).
• Use git to checkout and discard any changes to goldens that were not on your prediction list.
• Verify the newly written goldens manually to confirm the pixel adjustments are correct.

🤖 Agent takes the wheel — steps 1, 5, and 6: shrinking the repro, writing the invariant tests, and regenerating the predicted goldens. All three have crisp inputs and a mechanical definition of done.

🛡️ The check that makes it safe — the prediction from step 3 compared against the actual failing-golden list, plus git diff --stat on the golden directory. Any file changing that was not predicted stops the change dead; that comparison is what keeps -update from ratifying a bug.

Anti-Patterns to Avoid

• Using -update to Make the Suite Green: Ratifying the bug into the goldens destroys the utility of the test suite.
• Band-Aid Stacking: Writing logic to intercept and fix a bug downstream of where it occurred.
• Reimplementing Code in Diagnostic Scripts: Always run the real production code in your diagnostics.