ADR-0025: Verification Surfaces and Unversioned Runtime State

Field	Value
Status	Proposed
Date	2026-08-26
Author	BLU (lead architect)
Approver	Thomas Scola
Scope	Platform-wide — CI, deployment, runtime configuration, agent execution
Related	ADR-0024 (portable execution contract), ADR-0019 (execution context as a governed resource), ADR-0023 (repository authority model), ADR-0027 (supersedes Rule B.5)

---

Context

A single night of incident work across five subsystems produced two defect classes that recurred independently in tools that share no code, no vendor, and no purpose. Each was rediscovered from scratch several times because neither had a name.

This record names them, so the next occurrence is recognised rather than re-derived.

Class 1 — a verification surface that validates form while asserting a fact it never checked

Seven instances, in seven unrelated systems, in one night:

Surface	Reported	Actual
GitLab CI Lint	config valid	pipeline creation refused the same config; five days of zero-job pipelines passed invisibly
gc --dry-run	status=verified for 11 rigs	one rig's database did not exist; the real run failed on it
Repository search for runner_tag:	one component migrated	the query could not match runner_tags: — it excluded its own counter-evidence by construction
grep in a deleted worktree	zero forbidden constructs	the reader had no valid target; both constructs were present
gc doctor --fix	"fix attempted"	check still failing — an attempt reported as an action
--allow-unconfigured	would end a 9012-restart crash loop	leaves the damaged config intact; every failure signal goes quiet at once
Checksum of a path string	file verified	two files with different inodes answered to one path

The shape is constant. A check reports absence of a fault when its method could not have detected that fault. A passing result is indistinguishable from a result that was never computed.

Recurrence, 2026-09-02. A GitLab merge-request diff view is the same class of surface. For a branch cut before its target moved substantially, the compare view reports the full accumulated divergence as if it were the branch's own change — one instance showed a 4765-line conflicted diff whose actual unique content, found via the branch's own commit list rather than the raw compare, was a single 4-line addition. Check unique commits (git log branch..target, or an MR-commits listing filtered to non-ancestor commits) before concluding a change is large or genuinely conflicted.

The most dangerous variant is the one that appears resolved rather than merely applied: --allow-unconfigured would have silenced 9012 visible failures while the defect survived with no surviving indicator. Nothing subsequently fails to prompt a second look.

Class 2 — state that decides runtime behaviour and is invisible to every merge request

Four services, one shape:

• .gc/site.toml — rig path bindings exist only on the host. This is the entire reason bead hq-ztdb exists: a startup blocker that no repository contains.
• openclaw.json — gateway.mode, auth.mode, tailscale.mode, controlUi.allowInsecureAuth live only in a Docker volume. gateway.mode=remote reached that file on 2026-08-21 through a path nobody can name, five days before anyone noticed. Its own audit log records the byte trajectory of the change — 6826 → 7115 → 7169 → 7165 → 7171 → 7174 — which is someone authoring configuration by hand against a live service.
• .beads/config.yaml — issue_prefix reads du on the host and bc in the repository. Neither is wrong; there is no arbiter.
• compliance-copaw — pinned to a July image while its own :latest tag moved in August. The intended image sits unused on the same host.

In each case the source repository is complete, reviewable, and describes something that is not running.

Decision

A verification surface must be able to fail for the reason it is trusted to detect. State that determines runtime behaviour must exist in source and reach production only through a pipeline.

Rule A — falsifiability of checks

1. A gate that cannot be shown to fail is not a gate. Every gate carries a negative test that deliberately breaks its condition and proves non-activation. A gate accepted without a demonstrated failure is NOT_ESTABLISHED, not PASS.
2. DRY_RUN = STRUCTURAL_PREVIEW, never RUNTIME_ACCEPTANCE. A dry run reports shape. It does not report that the operation would succeed. LINT_PASS != CREATION_SUCCESS. DRY_RUN_PASS != TOPOLOGY_VALID.
3. A component contract change requires a consumer pipeline-creation canary, a fresh consumer pipeline, and creation-error capture. Never landed on lint alone — lint is insufficient by construction, not by policy.
4. A search that finds X cannot establish that not-X is empty. An absolute claim ("the only one", "none exist") drawn from a positive-match query is invalid. State the query's blind spot or do not make the claim.
5. Identify objects by content, never by name, size, or timestamp. Three separate times in one night, name-or-mtime reasoning selected the wrong artefact — including once where it would have discarded the only live ledger on the host, whose directory mtime read seven weeks stale.
6. Verify by outcome, not by exit code. A command reporting success while the condition persists is the default failure mode, not an edge case.

Rule B — no unversioned runtime state

1. Configuration that determines runtime behaviour is source-controlled and deployed by pipeline. Infrastructure and configuration are code. No manual host mutation — however carefully targeted, checksum-verified, dry-run, or diffed against a restore point.
2. A repair applied by hand is a defect even when it is correct, because it is invisible to review, unreproducible on a fresh deployment, and silently reverted by the next deploy.
3. Where a deploy step writes configuration, it must target the surface the service actually reads. A bind-mounted volume can expose a stale file at the volume path while the live file sits at the bind source. A deploy targeting the wrong one succeeds, reports green, and changes nothing.
4. Runtime artefacts are never tracked — databases, backups, ports, sockets, PIDs, locks, generated hooks and routes. Authored configuration is always tracked. Classify before removing; do not delete tracked files on the assumption they are generated. For .beads/, the upstream-tracked pair is exactly config.yaml and metadata.json (gascityhall/gascity engdocs/design/beads-dolt-contract-redesign.md); a file's own header claiming "canonical, git-tracked" is not upstream authority and must be verified against that source, not taken at face value — a blanket rig-level untrack has repeatedly conflated this pair with identity.toml and other non-canonical files.
5. Withdrawn 2026-08-26. Do not treat observed runtime as the specification when it disagrees with upstream documentation. The replacement rule is ADR-0027: upstream docs define the required architecture; upstream released code defines the supported implementation; runtime evidence measures conformance. Version skew is still a first-class fact — establish the deployed version and read source at the revision the artefact was built from — but a disagreeing binary is an upgrade, misconfiguration, or dirty-deploy signal, not a license to invent a third architecture. An OCI revision label still converts an unanswerable version question into a one-file read.

Rule C — corroboration and controls

1. Corroboration requires independent premises, not merely independent methods. Two lanes agreeing is not confirmation when they share an unexamined assumption. Before treating agreement as evidence, name what both checks assumed and verify that. Recorded because it happened: two lanes, two different tools, one shared belief about which branch was authoritative, reported as mutual confirmation of a severe defect that did not exist.
2. A positive control that shares the failed assumption confirms nothing. A control proving "the search index is live" while running against the wrong tree confirms the instrument and says nothing about the target. Control for what you are reading, not only for whether reading works.
3. Every check records the ref, host, or version it ran against. A result without its context is not reproducible and cannot be audited. In this estate the assumption to name is which branch is authoritative for a given repository — component work landing on release/v0.1.x while main carries a different tree is enough to invert a conclusion.
4. A gate must refuse to pass when it has nothing to check. A discovery-based test job that finds zero suites and exits 0 reports success for an empty set. The correct behaviour is to fail: "no suites found — this job would pass vacuously."
5. Naming what actually gates a merge is part of the merge record. Not "CI green" — which job, how long it ran, and whether it executed the code the change touches. Two projects surveyed in one night had merge evidence consisting of a syntax check that never loaded the changed code.

Consequences

Every gate the platform ships must now demonstrate its own failure before it is trusted. That is more expensive to build and it is the only thing that distinguishes a gate from a decoration.

The estate-wide CI standardisation programme inherits both rules directly: the pipeline-creation canary exists because lint could not fail for the right reason, and the profile model must cover runtime configuration declaration rather than only pipeline definition. A project whose .gitlab-ci.yml reaches near-zero while its runtime configuration still lives in an unversioned volume has moved the invisible state, not eliminated it.

Cost of not having these names: roughly four hours in one night, spread across seven rediscoveries of Class 1 and four of Class 2, plus two retracted findings and one authorised action withdrawn before execution. Each individual instance was diagnosed correctly; none was recognised as an instance of anything.

Both rules are self-applying, and that is deliberate. Rule A judges the tests written for Rule B. Rule B governs where the artefacts Rule A inspects are allowed to live. A verification surface exempt from its own standard is the first defect in this record.

Known limits. Rule A raises the cost of every gate; some low-risk checks will not justify a negative test, and that exemption must be stated where taken rather than assumed. Rule B has no answer yet for configuration a service writes about itself at runtime — legitimately dynamic state that cannot live in source. Naming that gap is preferable to a rule that quietly fails on it.

Amendment 2026-08-26. Rule B.5 ("the runtime contract wins over documentation") is withdrawn. ADR-0027 is the replacement: upstream documentation defines the target; upstream released code defines the supported implementation; runtime evidence measures conformance. Version identification remains required. Runtime disagreement is no longer a license to redesign.