Skip to content

ORACLE CANONICAL ARCHITECTURE

Authority: Gas City Docs (docs.gascity.com), OpenClaw Docs. Principle: Oracle is the production reference Gas City runtime. It consumes artifacts and provisions rigs on demand; it does not permanently host 136+ static development checkouts. Rule: Before creating any new directory, repository, runtime component, configuration file, deployment mechanism, or workflow — determine whether the capability already exists upstream. If it exists upstream, extend or configure it. If not, document why and define ownership, inputs, outputs, and lifecycle.

0. Operator Access & Separation of Environments

Invariant: OPERATOR_LOCATION != PRODUCTION_CITY_LOCATION — GitLab defines source. Oracle runs the production reference City. Mac runs the local development City. NAS provides durability/backup. Tailscale connects.

Layer Role
GitLab Authoritative source, CI/CD, packages, containers, MRs
Oracle (bluefly-platform.tailcf98b3.ts.net) Production Reference Gas City Runtime + Beads Work Authority
Developer Mac Local Development City (BluCity) + Thin Operator Client (gc, glab, bd, CMUX)
NAS (blueflynas.tailcf98b3.ts.net) Durability custodian, database backups, release artifacts, model storage — never hot runtime
Tailscale Private encrypted overlay network (MagicDNS, Tailscale SSH)

Operator Flow: 

device → Tailscale → bluefly-platform → blu / gc / bd / API

Clients do not require full local source trees or shadow Beads ledgers to operate the production Gas City.

1. Canonical Filesystem & Dedicated Storage

Dedicated Block Volume Mount (/opt/bluefly/ or /srv/gascity/bluefly)

To prevent root-disk exhaustion, Gas City hot runtime is mounted on a dedicated high-speed local block SSD.

/opt/bluefly/
├── blucity/                    # Gas City root (production deployment)
│   ├── city.toml               # Deployment policy (portable, committed)
│   ├── pack.toml               # Pack identity + imports (portable, committed)
│   ├── packs.lock              # Pinned import hashes (CLI-generated, committed)
│   ├── agents/                 # Agent directories
│   ├── formulas/               # Reusable workflow methods
│   ├── orders/                 # Scheduled/event-driven dispatch
│   ├── commands/               # gc subcommands
│   ├── doctor/                 # Custom gc doctor checks
│   └── .gc/                    # Machine-local state (NEVER committed)
│       ├── site.toml           # Rig path bindings + workspace_name
│       ├── credentials.toml    # Secret pointers only (0600)
│       ├── store/              # High-churn SQLite infra store (sessions, orders, nudges)
│       ├── worktrees/          # Ephemeral agent worktrees (auto-pruned on bead close)
│       └── runtime/            # Orchestrator sockets, locks, logs
├── deployments/
│   └── agent-docker/           # Infrastructure containers (Docker Compose)
└── runtime/
    ├── data/                   # Persistent database volumes (Dolt server data)
    ├── logs/
    └── secrets/                # Secrets at rest (chmod 700, never in Git)

Machine-Local Tool State

~/.gc/                          # User-global Gas City state (contexts.toml, auth keys, global cache)
/opt/bluefly/blucity/.gc/       # Machine-local site bindings & hot runtime

2. Rig Provisioning on Demand

Oracle does not permanently maintain static checkouts of all 136+ Bluefly projects.

ON-DEMAND PROVISIONING:
gc --context prod rig add --git-url https://gitlab.com/blueflyio/<repo> --name <name>
  • Rigs are provisioned dynamically from GitLab when work requires them.
  • Ephemeral worker worktrees are allocated under .gc/worktrees/<rig>/ during execution.
  • Lifecycle automation (auto_prune_worker_dir = true, auto_reap_closed_bead_worktrees = true) cleans up worktrees upon Bead completion.

3. Storage Architecture & Lifecycle Compaction

============================================================
ORACLE DISK PROTECTION & COMPACTION GATES
============================================================

1. WORKTREE PRUNING:
   auto_prune_worker_dir = true
   auto_reap_closed_bead_worktrees = true
   (Gated on clean git status, zero unpushed commits, and closed bead state).

2. DOLT AUTO-GC:
   auto_gc_enabled = true
   (Runs background Dolt maintenance to prevent noms journal bloat).

3. BACKUP RETENTION:
   Periodic cleanup of .beads/backup/ with bounded retention policy.

4. SPLIT STORAGE CLASSES:
   work      → Dolt (durable work ledger)
   infra     → SQLite under .gc/store (high-churn sessions, orders, nudges, graph)

4. Canonical Runtime Ownership

Layer Owner Managed by
Gas City supervisor upstream gc supervisor systemd
City config blucity GitLab repo GitLab CI (deploy:oracle)
Rig bindings .gc/site.toml (machine-local) gc-site-bind (CI) / gc rig add
Pack imports blucity-packs GitLab repo gc import install
Container services agent-docker GitLab repo Docker Compose
OpenClaw gateway agent-docker compose Docker Compose
Dolt database Gas City managed gc start
Beads (work store) Gas City managed gc start
Machine-local state Tool-owned ~/.gc/, blucity/.gc/site.toml
Secrets secrets/ + CI variables 1Password / GitLab CI variables

5. Canonical Artifact Flow

Source code (GitLab)
  → build → Docker image / NPM package / OCI image
  → push  → Registry (GitLab Container Registry / NPM)
  → pull  → Oracle (agent-docker compose)
  → run   → Runtime service

blucity-packs (GitLab)
  → gc import install (reads pack.toml + packs.lock)
  → Oracle pack cache
  → city loads → Agent behavior (prompts, formulas, orders, doctor checks)

6. Cold-Start Deployment

1. Provision Oracle host & attach dedicated block volume (iac / cloud-init)
2. Mount block volume at /opt/bluefly
3. Clone blucity → /opt/bluefly/blucity
4. Clone agent-docker → /opt/bluefly/deployments/agent-docker
5. Push blucity through governed CI: stamp → deploy:oracle → gc-site-bind → verify
6. cd /opt/bluefly/blucity && gc import install && gc start
7. cd /opt/bluefly/deployments/agent-docker && docker compose up -d
8. gc status && gc doctor

Operator access: connect via Tailscale to bluefly-platform, then use blu/gc/bd.

7. Recovery & Maintenance

gc supervisor status           # check supervisor
gc doctor                      # check health
gc reload                      # reload config (no restart)
gc restart                     # restart agents
gc stop && gc start            # full restart
gc status && gc doctor         # verify
gc import check && gc import install

Current State

What exists today. Separated from target/migration so that six months out it is obvious what was real. Tag everything; do not aspirational-wash.

Capability baseline

🟡 N = unmeasured (not zero). Authority-context resolution (blu work current) fails preflight locally and the canonical workspace root is not a Gas Town workspace. The runtime baseline is owned by the Capability Registry (Gas City + Oracle), readable only via gt / blu remote. See ADR-0002. First runtime work item: restore measurability of N, not write capability code.

Oracle — the 3-VM incident (Observed 2026-06-24)

VM State Disposition
VM B 150.136.74.174 🟢 the real production server (60+ services); never imported to Terraform ADOPT (terraform import)
VM A 109832 / 100.74.25.39 ⚪ empty shell created by a mis-aimed apply; cloudflared crash-looping TERMINATE
VM C 100.103.48.75 🟡 Feb duplicate; runs the live OpenClaw (:18789) migrate OpenClaw → VM B, then TERMINATE

claw.copaw.us is broken because the "social" Cloudflare tunnel targets a Tailscale hostname VM B cannot reach; a socat proxy on :18789 bridges to VM C. Immediate fix: point the tunnel at http://localhost:18789.

Oracle — single-instance runtime (Observed 2026-07-04)

The 3-VM incident is consolidated: OCI reports one instance bluefly-platform (shape VM.Standard.A1.Flex, lifecycle RUNNING), public 150.136.226.103, tailnet 100.64.56.113. It is offline on the tailnet (Tailscale: last seen ~1d). This is not a conflict — the VM boots, but the 2026-07-03 serial console shows tailscale up run with an empty auth key ("invalid key: unable to validate API key") because the oracle-authkey field is missing from the Tailscale 1Password item (op read exits 1). That missing field is the root cause of the offline node.

The same boot surfaced two cloud-init defects, both fixed in iac MR !71 → release/v0.1.x: - op read used the invalid --raw flag (op 2.34.1). The git clone of agent-docker/blucity-packs failed "HTTP Basic: Access denied"; the most likely explanation supported by current evidence is that the failing op read left DEPLOY_TOKEN empty — not independently verified during that boot. Fixed to --no-newline. - beads download used releases/latest/download/beads_linux_arm64.tar.gz → 404 (release assets are version-stamped). Pinned to v1.1.0 (beads_1.1.0_linux_arm64.tar.gz, arm64); URL now returns 200.

iac CI is green on release/v0.1.x + main. Remaining owner action to bring the node online: mint a Tailscale auth key (admin 2FA) and populate the oracle-authkey field; the fixed cloud-init then joins the node on the next apply.

Tooling (Observed)

gt, bd, blu, buildkit resolve on PATH. The only beads DB found locally is empty. Workspace is BluCity (not BluTown); hard-coded BluTown paths in some gateway code cause silent timeouts — a portability defect, not a stale path (ADR-0003).

Governance repo

🟡 bluefly-governance exists locally only; not authoritative until pushed to a GitLab remote (ADR-0001).

Oracle — GitLab Runner fleet drift (Observed 2026-07-30)

🔴 terraform/gitlab/runners.tf (iac) declares 6 runners with specific tags/concurrency/resource limits. Live Oracle (/opt/bluefly/agent-docker/.../config.toml, SSH-verified) had only 2 of 6 registered, concurrent=8 global with zero per-runner limit and zero Docker resource caps — directly matching independently-observed host resource-storm and CI gitleaks-stampede incidents. Root cause traced (not assumed): deploy:oracle (agent-docker .gitlab-ci.yml) runs docker compose up -d on every deploy but had no step touching config.toml — registration was a one-time manual gitlab-runner register outside CI, never reconciled since. Full inventory: iac terraform/gitlab/RUNNER-INVENTORY.md. Fix in flight, not yet applied: agent-docker MR!211. See target lifecycle in GitLab Runner Convergence Lifecycle.

Target State

Where infrastructure is going. ⚪ Planned unless tagged otherwise. No commands here — see Migration Plan.

Factory Law

Nothing is edited in production. Everything is rebuilt from authority.

[!CAUTION] Constitutional Invariants 1. Every persistent byte has exactly one authoritative owner. 2. Runtime may never become authority. 3. Every projection is reproducible from authority. 4. Every cache is disposable. 5. Infrastructure defines how execution is constructed. 6. Execution mutates state. 7. State never mutates infrastructure. 8. Secrets are retrieved, not stored. 9. Every deployment must be reproducible from authority. 10. Authority may be replicated. Ownership may not. 11. Every architectural exception requires an explicit documented justification.

Authority Decision Test

Every engineering change must answer these five questions before it is merged. If a proposal cannot answer these, it must be rejected: 1. Who owns this data? 2. Who is allowed to modify it? 3. Who consumes it? 4. Can it be regenerated? 5. If not, where is its authoritative home?

Terminology: Projection

Projection: A derived representation that may be deleted and regenerated entirely from authoritative sources. A projection must never become the system of record. (Examples: receipts, search indexes, compiled graphs, documentation sites, package artifacts, and caches are all projections).

Clean Plane & Authority Graph

Separating customer experience, operator orchestration, durable work, source delivery, and physical storage:

Human & Experience Plane (Owns Interface & Approval) * Drupal / AMCS / ContextControl.ai owns customer UI, Canvas, context governance, approval workflows, and site building.

Operator Plane (Owns Orchestration) * Gas City owns agent execution, session lifecycle, routing, formulas, orders, packs, and worktree lifecycles.

Work & State Authority (Owns Work Graph) * Beads (bd / Dolt) owns tasks, epics, blockers, dependencies, and priority.

Source & Release Authority (Owns Code & Artifacts) * GitLab / Upstream SCMs own source code, merge requests, CI/CD pipelines, package registries, and container images. * 1Password owns secrets (retrieved dynamically, never persisted in Git). * Cloudflare owns DNS, edge ingress, and tunnel routing. * Keycloak owns federated identity and authentication.

Custodian (Owns Durability & Backups) * Synology NAS acts as custodian for durable backups, long-term artifacts, models, and restore points.

Execution Runtime (Owns Hot Compute) * Oracle Runtime VM acts as the production reference Gas City host on dedicated fast block storage. * Developer Workstation (Mac) acts as a lightweight local development City and client interface.

Storage Class Division & Lifecycle Law

============================================================
STORAGE CLASS DIVISION
============================================================

HOT RUNTIME (Oracle Dedicated Block SSD / Mac Local SSD):
  • .gc/worktrees/             (Active execution trees)
  • .gc/runtime/               (Sockets, process locks, fifos)
  • .gc/store/                 (High-churn SQLite infra: sessions, orders, nudges)
  • .beads/dolt/               (Active Dolt database)

DURABLE STORAGE (Synology NAS / Offsite Backup / S3):
  • Database backups & snapshots (bd backup / Dolt dumps)
  • Long-term artifact archive
  • Model weights & large embeddings
  • OpenClaw shared durable workspace

RULE: NEVER run hot .gc/worktrees, SQLite, or Dolt over SMB/NFS.

The Factory Flow

flowchart TD
    subgraph HumanPlane["Human & Customer Plane"]
        Drupal["Drupal AMCS / ContextControl.ai"]
    end

    subgraph SourceGitLab["Source & Delivery (GitLab)"]
        GL[GitLab Source / CI / Packages / Registry]
    end

    subgraph OperatorPlane["Operator Control Plane (Oracle Gas City)"]
        GC[Gas City Controller]
        Worktrees[".gc/worktrees (Ephemeral local SSD)"]
        InfraStore[".gc/store (SQLite High-Churn Infra)"]
        DoltStore[".beads/ (Dolt Work Graph)"]
    end

    subgraph CustodianNAS["Durability & Backup Custodian (NAS)"]
        Backups[(Platform_Data_Persistent - Backups/Dumps)]
        Artifacts[(Platform_Artifacts - Releases/Receipts)]
        Models[(Platform_AI_Models - Weights/GGUF)]
    end

    Drupal -->|HTTP/SSE Connected Client API| GC
    GL -->|Provision Rigs on Demand via Git URL| Worktrees
    GC -->|Executes Work in| Worktrees
    GC -->|Infra State| InfraStore
    GC -->|Durable Tasks| DoltStore
    DoltStore -->|Periodic Snapshots & Dumps| Backups
    GL -->|Release Artifacts| Artifacts

1. NAS (Synology DS224+) — Durability & Backup Custodian

Each Shared Folder represents a bounded context with a distinct lifecycle, mapping directly to Synology features (Snapshots, Hyper Backup, ACLs).

Shared Folder Lifecycle Authoritative Owner Description
Platform_Source Immutable GitLab / Upstreams Cold source mirrors / archive.
Platform_Infrastructure Declarative IaC Compose files, Traefik config, systemd units.
Platform_Identity Secure 1Password / IdP Trust relationships, exported config, certs, realm definitions. Secret material remains authoritative in 1Password.
Platform_Data_Persistent Dynamic / Backup DB Dumps & Storage Durable State: Database backups (Postgres, Dolt dumps), Drupal uploads, Object storage. NOT live mmap database engines.
Platform_Data_Cache Ephemeral DBs / Caches Regenerable State: Temporary indexes, cold caches.
Platform_Packages Derived Release CI Pipelines Internal package distribution mirrors (npm, Composer, Docker).
Platform_Artifacts Derived Artifact CI Pipelines Release bundles, receipts, compiled exports, test reports.
Platform_Knowledge Curated Humans / Agents Engineering standards, documentation, research.
Platform_AI_Models Static Training Pipeline Foundation models, LoRAs, GGUF, ONNX, TensorRT, weights.
Platform_AI_State Dynamic Vector DBs / RAG Vector snapshots, RAG index dumps, conversation archives.
Platform_Operations Append-only System Logs, Metrics, Audit, Receipts, Deployment history, Backups.

Snapshot & Backup Policy (Synology Optimized)

  • Hyper Backup (Offsite/Cloud): Platform_Identity, Platform_Knowledge, Platform_Operations, and Platform_Data_Persistent backups.
  • Excluded from Hyper Backup: Platform_Data_Cache, Platform_Packages, Platform_Artifacts, Platform_AI_Models.
  • Btrfs Snapshot Replication: Enabled on namespaces that benefit from rapid rollback: Platform_Infrastructure, Platform_Identity, Platform_Source, Platform_Knowledge.

2. Oracle — Dedicated Production Gas City Host

  • One runtime authority (Gas City Host): a single Oracle VM (VM B), fully reproducible from IaC. The Oracle VM is fundamentally a Gas City host, not a raw unmanaged server.
  • Dedicated Block Volume: Oracle mounts a dedicated local block SSD (e.g. /opt/bluefly/blucity or /srv/gascity/bluefly) for Gas City runtime, eliminating system-disk bloat and avoiding SMB/NFS latency.
  • On-Demand Rig Provisioning: Rigs are provisioned via Git URL (gc rig add --git-url ...) when work requires them. Oracle does not host 136 permanent static checkouts.
  • Lifecycle & Disk Protection Gates:
  • auto_prune_worker_dir = true and auto_reap_closed_bead_worktrees = true prune ephemeral agent worktrees upon Bead closure after clean-git checks.
  • auto_gc_enabled = true on Dolt + periodic maintenance prevents journal ballooning.
  • Bounded retention on .beads/backup/ prevents unbounded local backup growth.
  • Split storage architecture isolates high-churn infra (graph, sessions, messaging, orders, nudges) into SQLite under .gc/store.
  • Strict 4-Layer Deployment Hierarchy:
  • Terraform: Owns the Machine / VM.
  • cloud-init: Owns Host provisioning (bootstrap, native dependencies, Tailscale).
  • Host (Systemd/Native): Owns upstream host components: gc, openclaw, dolt.
  • agent-docker: Restricted entirely to Infrastructure containers (observability, identity, drupal, databases).

GitLab Runner Convergence Lifecycle

Terraform (iac/terraform/gitlab/runners.tf)
  -> gitlab_user_runner resources + masked group CI variables (RUNNER_TOKEN_*)
  -> GitLab Deployment (agent-docker deploy:oracle)
       -> docker compose up -d
       -> register-gitlab-runners.sh    (renders config.toml from template + tokens, restarts)
       -> verify-gitlab-runners.sh      (structural diff: declared template vs. live config.toml)
       -> runner-convergence-report.md  (CI artifact, every deploy, 90-day retention)
  -> Oracle Runtime
       -> drift fails the deploy job (allow_failure: false) — no silent divergence

3. Developer Workstation (Mac)

  • The local Mac is a lightweight development environment containing the local dev City (BluCity) and client tooling (gc, glab, bd, CMUX).
  • Contains only active development rigs (e.g. ContextControl, AMCS, BluCity-Docs, BluCity-Packs).
  • Does not host 136 permanent repository checkouts or production authority.

Migration Plan

How we get from Current State to Target State. Governed by the Dual-Lane Execution Strategy: Lane 1 builds product immediately; Lane 2 converges the factory in parallel without blocking product delivery.

Dual-Lane Execution Model

┌────────────────────────────────────────┐  ┌────────────────────────────────────────┐
│        LANE 1: SHIP PRODUCT NOW        │  │       LANE 2: FACTORY CONVERGENCE      │
│                                        │  │                                        │
│ • Drupal AMCS site template            │  │ • Dedicated Oracle block volume        │
│ • ContextControl.ai ↔ Gas City Tool API│  │ • Worktree pruning & Dolt auto-GC      │
│ • Canvas agent site building           │  │ • Split storage migration evaluation   │
│ • Bead status visualization in Drupal  │  │ • Pack / Formula registry hardening    │
│ • Portable Customer Pack release       │  │ • Clean GitLab CI & IaC deployment     │
└────────────────────────────────────────┘  └────────────────────────────────────────┘

Phase 1: Oracle Gas City & Storage Convergence (Lane 2)

  1. Dedicated Block Storage: Attach and mount a dedicated high-speed block volume for /opt/bluefly/blucity or /srv/gascity/bluefly to isolate hot Gas City runtime from the VM root disk.
  2. Lifecycle & Compaction Gates:
  3. Configure auto_prune_worker_dir = true and auto_reap_closed_bead_worktrees = true in city.toml.
  4. Enable auto_gc_enabled = true on Dolt and schedule periodic compaction runs.
  5. Configure bounded retention on .beads/backup/ to eliminate unbounded backup bloat.
  6. Split Storage Architecture Evaluation:
  7. Test routing high-churn infra classes (graph, sessions, messaging, orders, nudges) to SQLite under .gc/store.
  8. Preserve durable work in Dolt/Beads ledger.
  9. On-Demand Rig Model:
  10. Cease maintaining 136 static repo checkouts on Oracle; provision rigs dynamically via gc rig add --git-url ....

Phase 2: Private Cloud Durability & Backups (NAS)

  1. Provision Durable Namespaces: Create and configure Platform_* Shared Folders in Synology DSM (Platform_Source, Platform_Infrastructure, Platform_Identity, Platform_Data_Persistent, Platform_Artifacts, Platform_AI_Models).
  2. Apply Backup Policies: Configure Btrfs Snapshots and Hyper Backup rules for durable state and backups (WAL, logical database dumps, release artifacts).
  3. Strict Non-Hot Rule: Explicitly prohibit running live mmap database engines or hot .gc/worktrees directly over SMB/NFS mounts.

Phase 3: Product Integration & Multi-Tenant Packaging (Lane 1)

  1. Drupal ↔ Gas City Connected-Client API: Wire ContextControl.ai / Drupal Tool API to Gas City's HTTP/SSE control surface for headless agent session execution and turn streams.
  2. Customer Factory Release: Package the reusable capability layer into Bluefly Packs, Formulas, Skills, Recipes, and Containers for repeatable single-tenant customer deployments.