Engineering Story: Multimodal Vision Sidecar Productionization, Candidate Preview Fidelity, and Release v0.1.9 (2026-10-25)

Second-Vendor Reviews:

• Audited by claude:sonnet:med (session story-reviewer-sonnet): Approved with corrections. Clarified test counts (19 tests across test_lifecycle.py and test_scan.py; 14 tests in test_scan.py), replaced mixed metric figures with explicit insertion/deletion breakdowns, corrected snapdex DisallowUnknownFields location (client.go:128), sourced the first-probe 502 race condition to developer rollout telemetry, grounded Section 6 turn and token numbers to sprint study data, and documented that Issue #222 per-pass latency was an architectural estimate.

1. Header & Context

• Date: 2026-10-25
• Baseline: Follow-up to docs/studies/2026-10-25-vision-runtime-embedding-remote-install-and-goself-bootstrap.md at commit fbf4395 (2026-10-25).
• Scope & Goals:
  1. Complete multimodal scanning capabilities in the vision sidecar (scripts/vision-sidecar/server.py) for snapdex indexing (Issue #222).
  2. Resolve remote service deployment preview fidelity and deploy updated sidecar payload to inference node x600 (Issue #223).
  3. Resolve live production indexing crashes caused by degenerate Florence-2 bounding boxes without breaking strict JSON schema consumers (Issue #224).
  4. Curate evergreen system documentation across architecture, operations, roadmap, and code layout.
  5. Cut and publish CLI release v0.1.9 across Codeberg and GitHub remotes.
• Baseline Comparison Metrics (fbf4395..HEAD):
  • Total commits: 33
  • Code & release commits (feat|test|refactor|chore(release)): 8
  • Docs/Studies/Issues commits: 25
  • Tickets closed: 3 (#222, #223, #224)
  • Release published: v0.1.9 (tag v0.1.9, commit ad8d8b7)
  • Diff outside issues/: 25 files changed, 1,434 insertions(+), 185 deletions(-)
    • Test suites: +688 lines (+371 inserted in test_scan.py, +317 inserted / -8 deleted in Go tests).
    • Sidecar implementation: +127 lines inserted, -30 deleted across server.py, composite_benchmark.py, and smoke.py.
    • Installer & host plumbing: +82 lines inserted, -42 deleted across install.go, host.go, and vision_runtime.go.
    • Evergreen documentation & sprint studies: +479 lines inserted, -14 deleted.
    • Website modernization: +53 lines inserted, -89 deleted in website/page.yaml (net reduction of 36 lines).
    • Metadata & config: +5 lines inserted, -2 deleted (version.yaml, version.go, go.mod, go.sum, docs/README.md).

2. Executive Summary

Earlier on 2026-10-25, lmcoder established clone-free remote execution (Issue #220), allowing binaries and embedded sidecar payloads to be provisioned over SSH without requiring local git checkouts on remote servers like x600. However, when consumer application snapdex attempted to index real photo libraries against the newly embedded vision sidecar, three operational deficiencies emerged in rapid succession:

1. Object Detection Sparsity (Issue #222): Standard Florence-2 <OD> only recognized 4 of 19 owner-annotated objects. The sidecar needed opt-in access to <DENSE_REGION_CAPTION>, <OPEN_VOCABULARY_DETECTION>, and <CAPTION_TO_PHRASE_GROUNDING>, unified with depth proximity.
2. Stale Remote Preview False-Negatives (Issue #223): Running lmcoder install --host x600 --dry-run reported that lmcoder-vision.service was unchanged (keep: binary, unit, config unchanged), despite the local binary embedding an updated sidecar payload. The preview was executing the stale binary already installed on x600, which was blind to the candidate binary's embedded files.
3. HTTP 500 Crashes on Degenerate Boxes (Issue #224): When snapdex re-indexed 545 photos with dense captions and 16 open-vocabulary labels, 20 photos crashed with HTTP 500 because Florence-2 emitted degenerate zero-width line boxes (x1 == x2), causing box_fields to raise ValueError and abort the entire scan (discarding embeddings and detected faces).

Over three sequential lean sprints, the system was productionized and deployed live to x600:

• Rich Region Detection: Extended server.py with opt-in flags (dense_captions, labels, grounding_phrase) and unified coordinate normalization (box_norm) and disparity pooling (proximity).
• Candidate-Faithful Remote Preview: Introduced previewCandidateInstall in host.go, streaming candidate binaries to trapped temporary files (mktemp, cleaned on EXIT) to evaluate candidate logic directly against live remote host state.
• Strict Schema-Preserving Degenerate Box Filtering: Architect claude:opus:med discovered that snapdex's strict Go client uses DisallowUnknownFields (snapdex/internal/vision/client.go:128); adding a dropped_regions field would have broken all 545 scans. The fix introduced pure helper model_box with subpixel clamping and zero-area drop, logging journal warnings without altering response schema keys.
• Release v0.1.9: Signed with Minisign, cross-compiled for 4 OS/architecture targets via GoReleaser, and published with attached release assets to Codeberg and GitHub.

3. What Worked Well

• Zero-Coding Host Orchestration & Architectural Refinements: The host maintained strict zero-coding discipline throughout all three sprints. In Sprint 223, codex:sol:med clarified that extraction plumbing was already functional and correctly pinpointed runInstallRemote's stale binary invocation. In Sprint 224, claude:opus:med prevented a catastrophic schema regression by checking snapdex/internal/vision/client.go:128 and identifying DisallowUnknownFields.
• AST-Compiled Zero-Dependency Python Unit Testing: Testing machine learning services typically requires multi-gigabyte PyTorch/Transformers virtual environments with GPU runtimes. By compiling server.py and composite_benchmark.py functions via Python's ast module and mocking torch tensor batches, the developer (codex:sol:med) built a comprehensive test suite (14 test methods in test_scan.py, bringing the sidecar suite to 19 tests across test_lifecycle.py and test_scan.py) that runs in ~16s on any standard developer machine.
• Trapped Temporary File Streaming (previewCandidateInstall): Rather than trying to teach an outdated remote binary about new local fingerprints, streaming the candidate binary to a trapped temporary remote file guarantees 100% preview fidelity against real remote systemd services and configs without mutating installed files.
• Automated Signed Release Pipeline: harnez release executed cleanly: reading version.yaml, running GoReleaser with GOWORK=off, signing SHA256SUMS with Minisign non-interactively, pushing tags to both origin and github, and uploading 6 assets via fj release.

4. Honest Post-Mortem (Failures, Bugs & Near-Misses)

1. The Stale Remote Preview Trap (Issue #223)

• Symptom: After embedding updated Python scripts into lmcoder, lmcoder install --host x600 --dry-run reported keep lmcoder-vision.service: binary, unit, config unchanged.
• Root Cause: runInstallRemote executed ssh x600 ~/.local/share/lmcoder/bin/lmcoder install --dry-run. Because it was a dry-run, no new binary had been uploaded; the command ran the existing remote binary, whose embedded assets matched the remote install-state.json.
• Risk: An operator relies on --dry-run to know what will change. When dry-run falsely reported keep, it appeared that embedded script change detection was completely broken, discouraging deployment.
• Fix: Implemented previewCandidateInstall to stream the candidate binary to a temporary remote executable, ensuring the preview is always evaluated by the candidate binary.

2. The Strict JSON Decoder Near-Miss (Issue #224)

• Symptom: Ticket #224 initially planned to add a dropped_regions counter to the scan response dictionary to report skipped degenerate boxes.
• Near-Miss: Snapdex uses Go's json.NewDecoder(resp.Body).DisallowUnknownFields() at snapdex/internal/vision/client.go:128. Had the developer added dropped_regions to the Python response, all 545 photo scans in snapdex would have failed immediately upon receiving the unexpected JSON key.
• Fix: Architect claude:opus:med caught the constraint during refinement. The team adopted a "log warning only" policy in the systemd journal with zero schema modifications.

3. Startup Race Condition in Service Probing (Issues #223 & #224)

• Symptom: In both Sprint 223 and 224, developer logs recorded that the very first live smoke.py probe attempt right after remote service restart returned HTTP 502 Bad Gateway.
• Root Cause: systemd transitions lmcoder-vision.service to active as soon as Python exec starts, but Florence-2 Base and SigLIP 2 take several seconds to load weights onto CUDA memory and open the FastAPI HTTP listener on port 8750. The reverse proxy returned 502 while waiting for the upstream listener.
• Remediation: The developer allowed service startup to settle before probing; subsequent probes passed consistently. Live verification harnesses must poll readiness or account for model initialization latency.

4. Unverified Latency Benchmarks in Ticket #222

• Limitation Disclosed: The ~350–400 ms per Florence-2 task pass documented in Issue #222 was an architectural estimate based on single-pass CUDA times from Issue #217. Because the local developer machine lacked the Florence virtual environment, live latency profiling was omitted during the developer turn and deferred to production observation.

5. Quality & Invariants Audit

6. Efficiency & Velocity Assessment

• Sprints Executed: 3 full lean sprints (222, 223, 224) in ~3 hours.
• Subagents Dispatched:
  • searcher-222 (claude:haiku:low): 1 turn, 195.8k tokens, rated 5/5.
  • dev-222 (codex:sol:low): 4 turns, 2.6M cached / 156.7k new tokens, rated 5/5.
  • advisor-223 (codex:sol:med): 1 turn (1m18s), 280.6k cached / 38.1k new tokens, rated 5/5.
  • dev-223 (codex:sol:low): 4 turns, 7.9M cached / 198.3k new tokens, rated 5/5.
  • architect-224 (claude:opus:med): 1 turn (1m10s), 253.6k tokens, rated 5/5.
  • dev-224 (codex:sol:med): 4 turns, 3.1M cached / 127.0k new tokens, rated 5/5.
  • story-reviewer-sonnet (claude:sonnet:med): 1 turn (43s), 141.1k tokens, rated 5/5.
• Zero Zombie Invariant: All spawned subagent sessions were rated 5/5 and deleted immediately upon milestone completion.

7. Key Learnings & Evergreen Upstream

1. Candidate-Driven Remote Dry-Runs (previewCandidateInstall): Never delegate a remote dry-run to a pre-existing remote binary when the deployment vehicle is a new candidate binary. Always stream the candidate to a trapped temporary file to evaluate live system state. Promoted to docs/LmcoderOperations.md.
2. Strict Consumer Decoders Constrain Server Schemas: When downstream consumers use DisallowUnknownFields, server-side operational metrics (like dropped item counts) must not be added to payload dictionaries without coordinated consumer updates. Logging to systemd journal is the correct out-of-band operational channel.
3. AST Testing for Embedded Script Payloads: Compiling target functions with ast.parse and injecting mock namespaces enables testing non-compiled embedded scripts (Python, shell) directly in CI without heavy runtime dependencies. Promoted to docs/CodeLayout.md.

8. File & Diff Summary

Key Files Modified & Created (fbf4395..HEAD)

• scripts/vision-sidecar/composite_benchmark.py: Added model_box with subpixel boundary clamping and zero/negative area rejection.
• scripts/vision-sidecar/server.py: Added opt-in Florence tasks (DENSE_REGION_CAPTION, OPEN_VOCABULARY_DETECTION, CAPTION_TO_PHRASE_GROUNDING), unified box_norm and depth proximity, and degenerate box filtering.
• scripts/vision-sidecar/test_scan.py: Created 371-line AST-compiled test suite covering request parsing, degenerate box handling across all tasks, and stock photo dimensions.
• scripts/vision-sidecar/smoke.py: Added CLI probe flags and response validation for dense and grounded region arrays.
• cmd/lmcoder/host.go & cmd/lmcoder/install.go: Implemented previewCandidateInstall streaming candidate binaries over SSH for accurate remote dry-runs.
• cmd/lmcoder/vision_runtime.go & vision_runtime_test.go: Atomic symlink payload rotation with failure-boundary safety tests.
• docs/LocalInferenceStack.md, docs/LmcoderOperations.md, docs/Roadmap.md, docs/CodeLayout.md: Curated evergreen documentation.
• version.yaml & version.go: Bumped to v0.1.9.
• website/page.yaml: Modernized feature cards with multimodal vision sidecar and goself library integration.