Gas City Incident Recovery Playbook¶
Status: Active Playbook
Authority: BluCity-Docs/Engineering-Standard/operating-model/factory-operating-contract.md
Upstream Reference: docs.gascity.com/runbooks/managed-city-endpoints
1. Scope & Symptoms¶
This playbook handles Gas City control-plane failures and store drift:
- gc start FATAL: deprecated city.toml [dolt] endpoint conflicts with canonical managed city config
- bd doctor: Dolt server unreachable at 127.0.0.1:0 (missing .beads/dolt-server.port)
- Supervisor stuck in retry loop or launchd service unresponsive
- Stale .gc/ runtime state created outside city root
2. Recovery Recipes¶
Scenario A — Deprecated [dolt] Section Conflict (gc start FATAL)¶
Root Cause: city.toml contains a legacy [dolt] section that conflicts with gc.endpoint_origin: managed_city in .beads/config.yaml.
Resolution:
1. Open city.toml in city root (BluCity/city.toml).
2. Remove the legacy [dolt] section entirely (lines specifying port, host, max_connections, etc.).
3. Restart supervisor via launchd (macOS):
launchctl bootout gui/$(id -u) "$HOME/Library/LaunchAgents/com.gascity.supervisor.plist"
sleep 2
gc start
gc doctor
bd dolt show
Scenario B — Missing Port Mirror (bd sees Port 0)¶
Root Cause: Managed Dolt server is running, but the compatibility mirror .beads/dolt-server.port was deleted or not generated.
Resolution: 1. Check running managed Dolt state:
cat .gc/runtime/packs/dolt/dolt-state.json
gc start while supervisor is running to trigger the normalization pass (syncConfiguredDoltPortFiles):
gc start
cat .beads/dolt-server.port
bd dolt show
Scenario C — Misplaced .gc Runtime Directories¶
Root Cause: gc commands were executed from workspace parent directory instead of city root, leaving .gc/promos or .gc/runtime outside BluCity/.
Resolution: 1. Identify orphan processes:
ps aux | grep 'blueflyio/\.gc'
promos/ into city root .gc:
mv [WORKSPACE-ROOT]/.gc/promos [WORKSPACE-ROOT]/BluCity/.gc/promos
.gc directory:
rm -rf [WORKSPACE-ROOT]/.gc