The Canary That Skipped Itself

Who this is for — anyone who has ever shipped a fix, watched its test go green, and had the bug come back anyway.

Read this if — you write regression tests that shell out to a real external tool.
Skip this if — every check you own runs in-process, with no environment it could be missing.

Takeaways

1. A regression test that can silently skip is not a regression check — it is a check-shaped hole.
2. Don’t infer an external tool’s version threshold from one installed copy; probe the binary that will actually run.
3. When a bug’s only symptom lives inside a GUI the agent cannot see, ask a human to look, one change at a time.

wayreel records terminal and desktop demos headlessly; conreel runs it inside a container so the same .reel script produces the same recording on any host. They are a small, real Solo Fleet pair.

This chapter is one debugging session across both repositories, and it is unusual for this book in one way: the agent was wrong twice, confidently, before it was right. Both wrong answers were tested against a real, running binary. Both shipped with a ✅ next to them. That is the point of writing it down.

On Evidence

What is real here. Every command, error message, commit hash, and file path below is taken from the conreel and wayreel repositories and can be checked with git show <hash> or by rerunning the command. The dialogue quotes are the actual session transcript, not reconstructed — paraphrasing would have lost the thing worth keeping: how ordinary each wrong turn sounded at the time.

1. A Symptom the Agent Cannot See

conreel watch reels/gedit-demo.reel ran clean — no errors, a 👀 Mirror window opened line, a normal exit. The human running it said:

“I saw no mpv window”

Nothing in the terminal output disagreed. That is the shape of the whole chapter: the bug’s only symptom lived inside a GUI window the agent has no way to observe. Piping stdout back and forth cannot answer “is there a window on your screen.”

Tracing the code instead of guessing from the silence found the defect in wayreel/mirror.go. mpv/ffplay/wf-recorder were launched with no Stdout/Stderr captured, and the success message printed the instant the process launched — not after it had connected to Wayland and mapped a window. A connect failure made the player exit almost immediately, and that failure was invisible: no error text anywhere, the CLI still reported success, the recording continued.

No check existed to catch this, and that absence was the bug: a launched process was being treated as a working one.

The fix captures both processes’ combined output, waits a 300 ms liveness window after launch before declaring success, and prints the captured output whenever the mirror closes for any reason other than the user closing it. The next run surfaced the real error immediately:

mpv: ../video/out/x11_common.c:679: vo_x11_init: Assertion `!vo->x11' failed.

mpv’s Vulkan/Zink probe was failing in a GPU-less container, and its own fallback chain hit an assertion bug in its X11 backend instead of degrading gracefully. Pinning mpv to --gpu-context=wayland --vo=gpu plus LIBGL_ALWAYS_SOFTWARE=1 sidesteps the probe entirely. Verified across all four reels in the repository — wayreel@7a78ca0.

🤖 Agent takes the wheel — reading mirror.go, scenario.go, and main.go across two repositories to trace exactly what a 👀 Mirror window opened message does and does not guarantee, then instrumenting the silence.

🛡️ The check that makes it safe — conreel watch reels/gedit-demo.reel, which no longer lies. The captured player output is printed on any unexpected close, and the liveness window turns an instant exit into a failure instead of a 👀 Mirror window opened line.

That part of the session is a clean, standard fix: instrument the silence, read the real error, fix the real cause. The next part is not.

2. Wrong Fix #1: The Threshold Was a Guess

With the mirror fixed, the user was asked to run each reel individually and report what the mirror actually showed — a deliberate protocol, not a formality, because the agent still could not see the window. After one run:

“it was a reel syntax error”

Wrong, but usefully wrong: a specific, checkable claim instead of “something looked off.” A screenshot of the mirror window followed, showing foot’s own startup line:

error: foot: /tmp/wayreel-foot.ini:1: [colors-dark]: invalid section name: colors-dark

Not a reel syntax error. A foot.ini section name that foot’s own parser rejected — on the container’s foot 1.25.0, inherited from wayreel’s generated theme config.

FootColorsIni() (reelang/types.go) already had version detection for exactly this, added in an earlier fix (wayreel issue #20): shell out to foot --version, use [colors-dark]/[colors-light] on foot ≥ 1.23, and the legacy [colors] section below that. The container runs 1.25.0 — past the coded threshold — so it should have worked. It didn’t.

The natural next move was to ask what [colors-dark] actually is to foot, rather than trust the threshold. The container’s own shipped default config, /etc/xdg/foot/foot.ini, defines only [colors]/[colors2], and foot --check-config on the container rejected [colors-dark] outright — on 1.25.0, a version the existing code believed already supported it.

From that one data point the conclusion looks inescapable: [colors-dark] was never real foot syntax at any version. The fix that followed dropped the dialect switch entirely and always emitted [colors]. It built, it passed go test ./..., and a full container rebuild confirmed the error was gone:

“looks clean now”

That confirmation was real. It was also not the end of the story.

3. Wrong Fix #2: The Feature Was Real, Just Newer

The next question — asked by the user, not found by the agent — is the hinge of this chapter:

“The colors issues was created after I saw the host foot complain that ‘colors’ was deprecated and I should use ‘colors-dark’. A warn when foot started. Now you say it never had any ‘colors-dark’ notion?”

That is a direct contradiction of the “never existed” conclusion, from a second, independent foot installation. Checked rather than argued about:

$ foot --version
foot version: 1.27.0 -pgo +ime +graphemes +toplevel-tag +blur -assertions

$ foot --check-config --config=/tmp/test-colors.ini    # [colors] section
warn: config.c:1635: /tmp/test-colors.ini:2: [colors]: deprecated; use [colors-dark] instead

$ foot --check-config --config=/tmp/test-colors-dark.ini    # [colors-dark] section
$                                                              # accepted, no warning

Both installations were telling the truth. The container’s foot 1.25.0 genuinely rejects [colors-dark]. The host’s foot 1.27.0 genuinely deprecates [colors] in favor of it. Foot’s own changelog gives the boundary: 1.26.0, released 2026-03-03, renamed [colors]/[colors2] to [colors-dark]/[colors-light].

So the feature was never fictional. The original threshold — 1.23 — was simply the wrong number, and “always emit [colors]” was a second wrong fix that would have quietly reintroduced, on any newer foot, the exact defect issue #20 existed to remove: a deprecation warning baked into every recording.

🧑‍✈️ Human holds the wheel — noticing that a confident, tested, “corrected” conclusion contradicted something they had personally seen on their own machine, and saying so instead of accepting the correction. Neither foot version was in the agent’s own environment; only a human checking a second real machine surfaced the contradiction.

4. Why Both Wrong Fixes Looked Verified

Both conclusions shipped with a ✅ beside them: 1.23 in wayreel/issues/20-foot-ini-colors-dark-compat.md, “never real syntax” in conreel/issues/05-foot-colors-dark-workaround.md. And reelang_test.go already had TestFootColorsIniAcceptedByFoot — a real test, running a real foot --check-config against the generated config, exactly the kind of check Canary-First Development argues for. It should have caught both mistakes on the first go test ./....

It didn’t, because of one line:

func TestFootColorsIniAcceptedByFoot(t *testing.T) {
    if _, err := exec.LookPath("foot"); err != nil {
        t.Skip("foot is not installed")
    }
    // ...
}

That guard is reasonable in isolation — don’t fail CI on a machine without foot. A second defect compounded it. A sibling test, TestFootColorsIniLegacyDialect, forced the legacy code path for its own assertions by calling footVersionOnce.Do(func(){}) — a sync.Once, spent, for the rest of that test binary’s life. Any later test wanting a genuine, fresh probe silently got the forced value instead. TestFootColorsIniAcceptedByFoot ran, didn’t skip, and reported green — while re-verifying whatever the previous test had forced, not the detection logic at all.

Two silent failures, stacked: a check that can no-op without saying so, and shared process state that one test could poison for every test after it. Both are easy to write by accident, and both produce exactly the same outward signal as a check that works — a green suite.

5. The Fix That Held: Stop Guessing, Start Probing

The threshold was wrong once at 1.23 and would be wrong again at any other hardcoded number the next time foot moves this boundary. FootColorsIni() now probes the installed foot binary at call time instead of encoding a version anywhere:

func detectFootColorThemeSupport() {
	f, err := os.CreateTemp("", "wayreel-foot-probe-*.ini")
	if err != nil {
		return
	}
	defer os.Remove(f.Name())
	if _, err := f.WriteString("[colors-dark]\n"); err != nil {
		f.Close()
		return
	}
	f.Close()

	out, err := exec.Command("foot", "--check-config", "--config="+f.Name()).CombinedOutput()
	footSupportsColorThemes = err == nil && !strings.Contains(string(out), "invalid section name")
}

Cached once per process, and defaulting to the legacy [colors] dialect — not the modern one — when foot is missing or the probe fails. [colors] stays valid, if deprecated, on every foot version: guessing the new dialect wrong breaks outright, guessing the old one wrong only warns.

The test fix mattered as much as the code fix. The poisoned sync.Once is now reset to a fresh state on every forced-dialect test’s cleanup, and TestFootColorsIniAcceptedByFoot resets it at its own start too, so it can never again inherit another test’s forced value.

🤖 Agent takes the wheel — writing the probe, rewriting the tests so a forced dialect can no longer poison a later real one, and rebuilding the container to exercise both foot 1.25.0 (legacy fallback) and foot 1.27.0 (modern dialect).

🛡️ The check that makes it safe — go test ./reelang/, where TestFootColorsIniAcceptedByFoot now runs a genuine probe every time it does not skip, plus a live conreel watch reels/smoke.reel on both foot versions, confirmed by the human watching the window (wayreel@7a78ca0, conreel@00db655).

6. The Rule This Session Earned

Neither wrong fix was a careless guess. Each was checked against a real, running foot binary, on a real machine, and confirmed clean. The mistake both times was treating one real environment as the real environment — and the regression test built to catch exactly that had been sitting there the whole time, never getting a chance to run honestly.

So the rule is not “test it.” It is this line, now in wayreel/AGENTS.md, outside the section harnez manages so it survives the next tooling sync:

Before marking an issue Resolved that depends on an external tool’s version-gated behavior, run its canary/regression test and confirm it didn’t skip — a skipped check is not a verified fix.

The companion habit is harder to write down, because no file can enforce it: when the only evidence of a bug is something a human can see and the agent cannot, the fastest path is not a better guess from the transcript. It is asking, one change at a time — and treating “looks clean now” as real evidence right up until a second, independent check says otherwise.

The same mistake wearing different clothes — one machine’s package set taken for every machine’s — is in The Single-Prompt Miracle and What It Cost §4, where a correct fail-closed fix made a headline feature unreachable on every distribution that does not ship pdftk.

Full history, including both wrong “Resolved” notes left in place rather than rewritten away, is in wayreel/issues/20-foot-ini-colors-dark-compat.md and conreel/issues/05-foot-colors-dark-workaround.md.