Canary-First Development

Who this is for — anyone whose feature depends on a tool, a kernel behaviour, or a sandbox they did not write.

Read this if — you have ever lost a day to an assumption about the environment.
Skip this if — you already probe every external tool in isolation before you build on it.

Takeaways

1. Verify the mechanism alone, in the real environment, before building anything on it.
2. The canary’s finding is the design input for the feature — write it down.
3. Never delete a canary; it is the cheapest regression check you will ever own.

Every project eventually bets on something it does not control: a CLI tool, a kernel behavior, a sandbox policy. Most debugging marathons are not caused by bugs in your code — they are caused by a wrong assumption about the environment, discovered three layers deep, at the worst possible time. Canary-first development moves that discovery to minute one.

A canary is a minimal, standalone verification script that validates one external mechanism before any feature code is built on top of it. It has no assertions framework and lives outside the main codebase validation suites. It runs against the real environment (filesystem, shell, external tool, network) and produces output you can inspect directly.

The name comes from the coal-mine canary: if the cheap verification check dies, you abort before investing in the tunnel.

When to Write a Canary

Write one whenever a feature depends on a mechanism that:

• Involves an external tool (mdbook, chromium, grim, tesseract, ffmpeg, curl, …)
• Relies on OS-level behavior (PTY redirection, process substitution, socket lifecycle, file descriptor inheritance)
• Requires a specific environment (Wayland compositor, Xvfb display, GPU vs software render, snap packaging restrictions)
• Reads back its own output (tee capture, screenshot OCR, log scraping)

Rule of thumb: If you cannot verify it with a fake, write a canary before writing the feature.

🤖 Agent takes the wheel — writing the canary. It is short, standalone, throwaway-shaped, and its success criterion is visible in the output. This is the single best first task to hand an agent on a new integration.

🛡️ The check that makes it safe — make canary. It runs the real tool against the real filesystem and exits non-zero on the first surprise, so a wrong assumption costs thirty seconds instead of a weekend.

The Pattern

mechanism alone → observe output → assert manually → then build

graph LR
    M[Isolate the mechanism] --> O[Run & observe in the real environment]
    O --> D{Behaves as assumed?}
    D -->|no| F[Document the finding & adapt the design]
    F --> M
    D -->|yes| B[Build the feature on solid ground]
    B --> K[Keep the canary as a regression check]

1. Isolate the mechanism: Write the smallest possible script, reel, or program that exercises only the external piece. No application logic, no abstractions.
2. Run it and observe: Check the output file, log, or terminal directly. Does the mechanism do what you assumed?
3. Document what you found: Note accuracy, failure modes, edge cases. This becomes the design input for the real feature.
4. Keep the canary: Don’t delete it. It is the cheapest future regression check and the fastest way for the next person to understand the mechanism.

Example — Snap Confinement (Our Own Canary)

During the creation of this book’s PDF generation pipeline, we wrote a canary (scripts/canary.sh) that ran chromium --headless --print-to-pdf to verify that we could print our built book index HTML.

The Finding: The canary failed with: Failed to write file /tmp/tmp.XXXX/test.pdf: No such file or directory

Because Chromium is installed via Snap, it runs in a sandboxed mount namespace. It could not write to the OS-level /tmp directory. However, it had access to the user’s home folder.

The Fix: We updated the script to write its temporary build outputs inside /home/uwe/projects/books/.tmp/. The canary passed immediately. This finding saved us hours of debugging inside the main application code!

Notice what didn’t happen: nobody stepped through the PDF pipeline with a debugger, nobody blamed mdBook, nobody filed a bug against Chromium. A thirty-line script found the sandbox boundary before the feature existed — and it still guards that boundary today, every time make canary runs.

🧑‍✈️ Human holds the wheel — deciding what the finding means. “Chromium cannot write /tmp” is an observation; “therefore every generated artefact in this project lives under the repository root” is a design rule, and it has consequences an agent is not positioned to weigh.

The sequel proves the point better than the original. Years later the same project ran on a different distribution with a flatpak Chromium behind a shim — different vendor, different sandbox technology, same class of failure, caught by the same script. The full path from this finding to a shipped feature is walked step by step in The Golden Path.

Not every canary stays this honest. The Canary That Skipped Itself walks through one that silently t.Skip()ed instead of failing when its tool wasn’t on PATH — and let two wrong “fixes” ship as Resolved before anyone noticed the check had never actually run.