Skip to content

BLU Drupal Fleet Observability

Product Specification

Owner: Bluefly / OtterMon
Status: Active architecture refinement
Last Updated: 2026-09-13

Executive Summary

BLU Drupal Fleet Observability is the Drupal fleet-observability capability of the Bluefly Site Factory and AMCS architecture.

Customer promise: Know which Drupal sites need attention, why they need it, what work should happen next, and whether the resulting change was actually verified.

The capability operates across the authoritative Bluefly operational loop:

"Drupal sees it → Beads records it → Gas City works it → GitLab delivers it → Ottermon proves it"

  • Drupal sees it: Drupal modules, telemetry hooks, Drush commands, and the ContextControl UI observe site state and surface structured telemetry and findings.
  • Beads records it: Beads creates and manages durable work items, dependencies, blockers, and status.
  • Gas City works it: Gas City coordinates governed remediation work through agents, formulas, and sessions within governed worktrees ($WORKSPACE_ROOT).
  • GitLab delivers it: GitLab owns source code, merge requests, CI pipelines, releases, container/package registries, and deployment evidence.
  • Ottermon proves it: OtterMon ingests runtime telemetry, audits events, and produces structured verification proofs confirming the intended invariant holds.

This product is not another workflow engine, agent router, deployment system, or source of work authority.

Product Principles

1. Drupal-native first

Use Drupal APIs, Drush, Composer, configuration entities, Tool API, and established contrib patterns before adding custom external machinery.

The Drupal implementation should be useful as a normal Drupal project even when Gas City is not connected.

2. Observe once, reuse everywhere

Checks, behaviors, telemetry ingestion schemas, findings, and evidence must be structured and reusable rather than implemented as one-off customer code.

3. Findings become governed work

An observation is not automatically a fix.

A finding produces or updates a Bead. Gas City routes remediation work. GitLab carries the source change and CI verification. Runtime verification by OtterMon closes the loop.

4. Mechanism success is not outcome success

A collector run, successful API call, green pipeline, or merged MR does not prove the Drupal site is healthy. The resulting runtime state must be re-observed by OtterMon against the intended invariant to produce cryptographic/ledger-backed proof.

Architecture

Drupal Site ("Drupal sees it")
  |
  | Telemetry Ingestion / OtterMon checks / Drush / Drupal APIs
  v
Structured Observation & Event Auditing
  |
  +--> Drupal / ContextControl UI
  |
  +--> Fleet aggregation / telemetry reporting
  |
  +--> Finding ("Beads records it")
          |
          v
        Beads (Work Graph / Dependencies / Blockers)
          |
          v
       Gas City ("Gas City works it")
          |
          v
     Agent / Formula Execution (Governed Worktree: $WORKSPACE_ROOT)
          |
          v
        GitLab ("GitLab delivers it")
     MR -> CI -> Release -> Deployment
          |
          v
    Drupal Deployment
          |
          v
     OtterMon ("Ottermon proves it")
     Telemetry Ingestion -> Invariant Re-observation -> Ledger / Evidence Verification
          |
          v
    Evidence / Closure Receipt -> Closes Bead

Components and Authority

Component Authority Responsibility
Drupal OtterMon module Bluefly Drupal contrib project Drupal-native checks, observations, configuration, Drush integration, telemetry hooks, and site-local evidence.
DDEV OtterMon add-on agent-platform/ddev-addons/ddev_ottermon Local development integration for OtterMon.
ContextControl / Drupal ContextControl product Human review, governed context, fleet/operator UI, and remediation visibility.
Gas City Upstream Gas City + Bluefly Packs Dispatch, sessions, formulas, events, and execution mechanics in $WORKSPACE_ROOT.
Beads Beads Durable remediation work graph, dependencies, blockers, and receipts.
GitLab GitLab + gitlab_components Source, MR, CI, packages, releases, deployment provenance, and built-in OpenTelemetry observability backend (traces, metrics, logs — group blueflyio, endpoint 87749026.otel.gitlab-o11y.com).
OtterMon OtterMon Platform Telemetry ingestion, event auditing, fleet-wide policy evaluation, and evidence verification / proof generation.
Studio UI agent-platform/tools/studio-ui Shared UI/component system for React and Drupal Canvas/SDC surfaces.
Dragonfly dragonfly/dragonfly Shared verification where a reusable testing layer is warranted.
Cedar / ContractPlane cedar-policies + ContractPlane Authorization and governed policy/contract evaluation where required.

No component should duplicate another component's authority.

Drupal OtterMon Direction

The old thin-collector-only design is no longer sufficient as the complete product definition.

The Drupal module should own the Drupal-native observability contract and expose reusable checks/behaviors as first-class configuration where appropriate.

Core responsibilities:

  • site/environment identity
  • Drupal core/module/theme posture
  • configuration posture
  • cron and queue health
  • recent error/watchdog signals
  • dependency/composer posture
  • update/security status
  • environment-specific policy checks
  • structured findings and evidence
  • Drush execution
  • machine-readable output for external consumers

The module should avoid owning:

  • agent orchestration
  • workflow state
  • deployment pipelines
  • general-purpose telemetry storage
  • customer-specific one-off remediation code

Those belong to existing factory authorities.

Fleet Views

The human/operator UI should support:

  • fleet health summary
  • site list by environment/status
  • per-site detail
  • findings worklist
  • severity/category filters
  • multisite awareness
  • observation timestamps
  • evidence/receipt links
  • remediation work status from Beads
  • GitLab MR/pipeline/release status where relevant
  • re-verification state after remediation

The UI should reuse Studio UI and Drupal Canvas/SDC patterns rather than create a separate design system.

Initial Check Families

The exact rule inventory should be versioned and evidence-backed, not frozen here as a permanent list.

Security

  • Drupal core security/update posture
  • contrib security/update posture
  • unsupported/EOL runtime components
  • risky production-only modules/configuration

Configuration posture

  • cache/aggregation production posture
  • trusted hosts
  • error reporting
  • development modules enabled in production
  • logging posture
  • critical site/runtime settings

Dependency and module health

  • unmanaged Composer dependencies
  • deprecated/abandoned components
  • dependency conflicts
  • incompatible Drupal/PHP/database combinations

Operational health

  • cron/queue health
  • recent error rate
  • watchdog/database-log growth where applicable
  • update-status availability
  • basic site availability where the deployment provides that signal

Performance

Performance checks should use measured evidence and avoid duplicating dedicated APM/observability platforms.

Finding-to-Work Contract

A finding should contain enough structured information to create or correlate work without embedding workflow logic in the Drupal module.

Minimum conceptual fields:

  • site/environment identity
  • check/rule identity
  • severity
  • observed value
  • expected invariant
  • evidence timestamp
  • evidence source
  • remediation guidance or reference
  • correlation/fingerprint
  • re-verification requirement

Operational Flow:

1. Drupal sees it:
   -> Site state changes or violation triggers observation / telemetry event
2. Beads records it:
   -> Correlates existing Bead or creates a new tracked work item
   -> Records blockers, severity, dependencies, and target invariant
3. Gas City works it:
   -> Routes remediation task to agent/formula execution
   -> Runs within isolated, governed worktree ($WORKSPACE_ROOT)
4. GitLab delivers it:
   -> Generates Merge Request (MR) -> CI pipeline gate -> semantic release
   -> Deployment to target environment
5. Ottermon proves it:
   -> Ingests post-deployment telemetry and audits runtime event stream
   -> Re-observes invariant to produce verification proof / receipt
   -> Bead closes only upon successful verification proof

Chat transcripts are not work authority.

Telemetry Ingestion, Event Auditing, and Evidence Verification

OtterMon operates as the verification authority across the fleet:

  • Telemetry Ingestion: Continuously collects structured metrics, watchdog signals, configuration diffs, and health beacons from Drupal instances across staging and production.
  • Event Auditing: Maintains an immutable audit trail of observation events, check executions, and environment state transitions.
  • Evidence Verification: Validates that code/config deployments have successfully satisfied the required invariants before work items are marked complete, generating signed cryptographic receipts.

External OtterMon Integration

The earlier specification assumed an external OtterMon platform with an ingest API, token provisioning, rule engine, and dashboard.

Treat those capabilities as integration assumptions until current implementation is verified.

Potential integration contract:

  • observation ingest
  • site/environment identity
  • finding/rule payloads
  • per-site or per-customer authentication
  • rate limiting
  • trigger/rescan
  • evidence/receipt retrieval

Do not duplicate an external capability in Bluefly merely because an integration is incomplete. Conversely, do not claim an external endpoint, dashboard, or token workflow exists until verified.

Deployment Model

Customer isolation

Customer deployments should preserve tenant and credential isolation.

A customer may have:

  • one or more Drupal codebases
  • production/staging/development environments
  • Drupal multisite installations
  • a dedicated or appropriately isolated Gas City/Beads context when agent execution is enabled
  • GitLab-backed delivery
  • customer-owned credentials/secrets where required

The observability architecture must not require all customers to share one mutable control-plane state.

Portability

The product must be deployable as part of the same portable Bluefly factory release to supported cloud/server targets.

The Drupal module remains Composer-managed. Runtime/operator services are deployed through the governed Agent Docker + IaC path, not workstation-local assumptions.

Development Integration

Local Drupal development should use the existing DDEV integrations rather than one-off host scripts:

  • DDEV Gas City
  • DDEV Agent BLU
  • DDEV OtterMon

Inside a DDEV environment, project worktrees use the project/container convention defined by the factory ($WORKSPACE_ROOT). Outside DDEV, Gas City owns worktree/session mechanics.

Durable documentation must not encode personal workstation paths or machine-specific directories. All paths are resolved dynamically relative to $WORKSPACE_ROOT.

MVP

The first useful product should prove the complete loop, not maximize rule count.

Required

  • OtterMon Drupal module enabled on a Drupal 11 site
  • a small curated set of high-value checks
  • machine-readable findings
  • fleet/site display in Drupal/ContextControl or the verified existing fleet UI
  • finding correlation into Beads
  • Gas City-routed remediation on one real defect
  • GitLab MR/CI evidence
  • re-observation after deployment
  • receipt showing the final invariant passed

Not required for MVP

  • a new workflow engine
  • a new agent router
  • a new telemetry database
  • a second design system
  • dozens of speculative rules
  • automatic production remediation without governance

Success Metrics

Bluefly

  • reusable Drupal contrib implementation
  • reusable check/rule library
  • no customer-specific forks for normal fleet behavior
  • findings can enter the standard Gas City/Beads/GitLab factory
  • observable reduction in duplicated monitoring/remediation logic

Customer

  • can see which Drupal sites require attention
  • can see evidence explaining why
  • can see remediation progress
  • can prove the resulting state was re-verified
  • does not need a separate bespoke Bluefly monitoring stack for every site

Factory

  • observation -> work -> source -> CI -> deployment -> re-observation is traceable
  • work state remains in Beads
  • source state remains in GitLab
  • agent execution remains in Gas City
  • customer/operator interaction remains in Drupal/ContextControl

Near-Term Work

  1. Confirm the current OtterMon Drupal module architecture and replace any remaining thin-collector assumptions.
  2. Define the first small check/behavior configuration model.
  3. Define the structured finding/evidence contract.
  4. Verify which external OtterMon platform capabilities currently exist before depending on them.
  5. Integrate one finding with Beads/Gas City.
  6. Surface work/remediation state through Drupal/ContextControl.
  7. Reuse Studio UI/Canvas/SDC for shared presentation.
  8. Prove one end-to-end remediation and re-verification flow.
  9. Add checks only from evidence-backed customer/fleet needs.

Authority Boundaries

  • Drupal/OtterMon module observes Drupal.
  • ContextControl/Drupal presents and governs human interaction.
  • Gas City operates agents and execution.
  • Beads owns work.
  • GitLab owns source and delivery evidence.
  • ContractPlane/Cedar governs contract/policy decisions.
  • Studio UI owns reusable visual components.
  • Agent Docker/IaC owns portable runtime deployment.
  • BluCity-Docs owns Bluefly-wide product doctrine.

If a proposed feature belongs to one of these authorities, extend that authority rather than creating a parallel implementation.

GitLab Observability Backend

Status: PROVEN_DEPLOYED — Pipeline CI/CD telemetry is already being collected automatically for the blueflyio group. No external APM vendor required.

GitLab Observability is the built-in application monitoring backend included with the blueflyio GitLab group. It is an alternative to Datadog, New Relic, and Dynatrace. Instrument with OpenTelemetry to send traces, metrics, and logs, then explore them alongside commits, pipelines, and issues.

Group dashboard: https://gitlab.com/groups/blueflyio/-/observability/dashboard

OTel Ingestion Endpoints

Transport Endpoint Notes
HTTPS (recommended) https://87749026.otel.gitlab-o11y.com:14318 Encrypted; use for production
gRPCS (recommended) https://87749026.otel.gitlab-o11y.com:14317 Encrypted; use for production
HTTP (non-TLS) http://87749026.otel.gitlab-o11y.com:4318 Dev/internal only
gRPC (non-TLS) http://87749026.otel.gitlab-o11y.com:4317 Dev/internal only

All endpoints accept traces, metrics, and logs.

Firewall: Allow outbound TCP on ports 4317–4318 and 14317–14318 to 87749026.otel.gitlab-o11y.com.

MCP Server

An MCP server is available to query observability data with natural language from AI assistants.

Endpoint: https://87749026.mcp.gitlab-o11y.com/mcp
Auth: Observability API key
Docs: https://gitlab.com/help/operations/observability/mcp_server.md

Resource Attributes

Configure the OTel SDK with these attributes to link telemetry to GitLab project, commits, and environments:

Resource Attribute GitLab CI/CD Variable Purpose
gitlab.project.id CI_PROJECT_ID Links telemetry to the GitLab project. Required for Duo integration.
gitlab.project.name CI_PROJECT_NAME Human-readable project name in dashboards.
service.version CI_COMMIT_SHA Correlates traces and errors to the exact deployed commit.
deployment.environment.name CI_ENVIRONMENT_NAME Environment tag (e.g. production, staging).

service.version and deployment.environment.name are OTel semantic conventions. gitlab.* attributes use the GitLab vendor namespace.

CI/CD Pipeline Export

Set the GITLAB_OBSERVABILITY_EXPORT CI/CD variable in group CI/CD settings to a comma-separated list of signal types to export from pipelines:

GITLAB_OBSERVABILITY_EXPORT = metrics,logs,traces

Application Instrumentation (Drupal/PHP)

PHP/Drupal applications should instrument via the OTel PHP SDK or the Drupal OtterMon module's telemetry hook, pointing at the HTTPS endpoint above. Set the resource attributes from CI/CD variables.

// Minimal OTel PHP bootstrap
$transport = (new \OpenTelemetry\SDK\Common\Export\Http\PsrTransportFactory())->create(
    'https://87749026.otel.gitlab-o11y.com:14318/v1/traces',
    'application/json'
);

For Ruby services (e.g. Gas City supervisor), use:

# Gemfile
gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'

# config/initializers/opentelemetry.rb
OpenTelemetry::SDK.configure do |c|
  c.resource = OpenTelemetry::SDK::Resources::Resource.create(
    'service.name'                  => ENV['CI_PROJECT_NAME'] || 'my-service',
    'service.version'               => ENV['CI_COMMIT_SHA'],
    'deployment.environment.name'   => ENV['CI_ENVIRONMENT_NAME'],
    'gitlab.project.id'             => ENV['CI_PROJECT_ID'],
    'gitlab.project.name'           => ENV['CI_PROJECT_NAME']
  )
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: 'https://87749026.otel.gitlab-o11y.com:14318/v1/traces'
      )
    )
  )
end

Endpoint Verification

curl -X POST https://87749026.otel.gitlab-o11y.com:14318/v1/traces \
  -H 'Content-Type: application/json' \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"spans":[{"traceId":"0000000000000000a1b2c3d4e5f67890","spanId":"a1b2c3d4e5f67890","name":"test-span","kind":1,"startTimeUnixNano":"1789618641320220160","endTimeUnixNano":"1789618641320220160","attributes":[]}]}]}]}'

Integration with Bluefly Architecture

  • Pipeline health is already collected automatically — no action required for CI/CD telemetry.
  • Application traces require OTel SDK instrumentation in each Drupal site / service. The deployment.environment.name attribute links traces to GitLab environments.
  • Duo integration requires gitlab.project.id in the resource attributes — enables AI-assisted trace analysis.
  • OtterMon should route structured findings into this backend as traces/spans where the finding has a request-level lifecycle, keeping fleet-wide event data in the same platform as source/pipeline data.
  • Gas City events can be bridged via an OTel span exporter on the event log, correlating factory execution to GitLab pipeline timelines.

References