swarm-house/docs/11-cicd-delivery.md

11 — CI/CD & delivery

How software, models, and configuration reach the fleet — reproducibly, scanned, versioned, and atomically rollback-able. Everything lives inside the air gap.

Decisions: ADR-0006 — IaC boundaries & fleet manifest, ADR-0007 — non-root image.

Source and pipelines: GitLab

GitLab (self-managed, on-prem) is the backbone: repositories, CI/CD, and the container registry in one system.

• Shared pipeline templates — one template library (ci-templates repo) defines the standard stages; service repos include and parameterize them instead of copy-pasting YAML.
• Multi-arch by default — every image builds for linux/amd64 and linux/arm64 via buildx on dedicated runners; GPU-inference images additionally build against the vendor's L4T-class base for the ARM targets.
• Runner fleet on ground k3s — build runners (amd64 + arm64), a GPU runner for inference smoke tests, and simulation runners that execute virtual-swarm regression scenarios (06).

graph LR
    subgraph gitlab [GitLab, on-prem]
        SRC["service repos<br/>+ ci-templates"]
        CI["CI pipelines<br/>build · test · scan"]
        REG["GitLab Container Registry<br/>images + generic packages"]
    end
    subgraph quality [Quality gates]
        SQ["SonarQube<br/>code quality"]
        TR["Trivy<br/>image + dependency scan"]
        SIMT["virtual swarm<br/>regression scenarios"]
    end
    subgraph delivery [Delivery]
        MAN["fleet release manifest<br/>semver, digests pinned"]
        MIR["registry mirror<br/>base station"]
        DOCK["dock: verify + apply"]
    end
    SRC --> CI
    CI --> SQ
    CI --> TR
    CI --> SIMT
    CI --> REG
    REG --> MAN
    MAN --> MIR
    MIR --> DOCK

Artifacts: GitLab Registry as the single store

One registry for everything, next to the pipelines that produce it:

Artifact	Stored as
Service images (multi-arch)	Container registry, immutable tags + digests
Detection model weights (YOLO-like family — architectures and weights vary per mission and improve over iterations)	Generic package registry, semver-versioned, checksummed
Dataset schemas	Generic packages, semver (03)
Compose bundles, radio profiles, provisioning configs	Generic packages, semver

Registry hygiene is part of the design: cleanup policies per repository, immutable release tags, access split between CI (write) and mirrors (read).

Consolidating on the GitLab registry removes a separate artifact platform from the stack — one fewer system to run inside the air gap, one auth domain, artifacts adjacent to the pipelines that build them. Scanning moves to Trivy (below).

Quality gates

Gate	Tool	Blocks merge when
Code quality, coverage, static analysis	SonarQube	Quality gate red
Image and dependency vulnerabilities	Trivy (in CI + scheduled re-scan of released images)	Critical findings without an accepted waiver
Behavioral regression	Virtual swarm scenarios — canonical seeds replayed, Parquet outputs asserted (row counts, coverage, staleness budgets)	Any budget exceeded

Scheduled Trivy re-scans matter in an air gap: a released image that was clean in March may carry a known CVE by June; the scan flags it for the next fleet release even though the image never changed.

The fleet release manifest

The central delivery idea: the fleet has exactly one version.

# fleet-release: 3.4.1
schema_version: 1
images:
  sensor-ingest:    registry.internal/fleet/sensor-ingest@sha256:9f2c…   # 2.1.0
  video-analytics:  registry.internal/fleet/video-analytics@sha256:5e11… # 3.0.2
  parquet-writer:   registry.internal/fleet/parquet-writer@sha256:aa04…  # 1.8.0
  state-publisher:  registry.internal/fleet/state-publisher@sha256:c7d9… # 1.4.3
models:
  detector: { package: detector-weights, version: 5.2.0, sha256: "e3b0…" }
schemas:
  telemetry: 2.1.0
  detections: 1.3.0
  state: 1.2.0
config:
  compose-bundle: 3.4.0
  radio-profile: production-2
  peer-registry: fleet-42-r7      # provisioning + revocations, see [05 — Network & security](05-network-security.md)

• Everything is semver-versioned individually, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
• Images are referenced by digest; docks verify signatures and checksums before applying.
• Rollback is atomic: re-apply the previous manifest. No per-service drift, ever — a drone either runs release 3.4.1 in full or 3.4.0 in full.
• The manifest is what the simulation farm certifies: regression scenarios run against the exact manifest that will ship.

Delivery to the drones

1. CI publishes a release manifest; the base-station registry mirror pulls all referenced artifacts inside the air gap.
2. Docked drones fetch the manifest, verify digests/signatures (05), stage the new Compose bundle, and switch on the next boot cycle.
3. Never mid-flight. Updates are a dock-only operation by construction — the update endpoint does not exist in the flight radio profile.

GitOps on the ground segment

The fleet release manifest handles drones (atomic, digest-pinned, dock-only). The ground k3s segment uses a different tool: Flux reconciles long-lived configuration from git — dashboard bundles, Prometheus rules, offload schedules, policy labels — without reprovisioning PVCs or re-running full Terraform applies for every label change.

graph LR
    subgraph git [Git, on-prem]
        REPO["service repos"]
        GITOPS["infra/gitops/<br/>ground overlay"]
    end
    subgraph ground_k3s [Ground k3s]
        FLUX["Flux controllers"]
        WH["warehouse workloads<br/>(Terraform-provisioned)"]
        CFG["policy ConfigMaps<br/>dashboard bundles"]
    end
    subgraph drones [Fleet]
        MAN["fleet release manifest"]
        DOCK["dock: verify + apply"]
    end
    REPO --> CI
    GITOPS --> FLUX
    FLUX --> CFG
    MAN --> DOCK

Segment	Delivery mechanism	Reconciler in production?
Drones	Fleet release manifest via registry mirror	No — no API server mid-flight; dock applies once
Ground k3s (warehouse, dashboards, CI runners)	Terraform for shape + Flux for ongoing config	Yes — stationary cluster, wired network, git is the source of truth
Dev simulation (k3d)	Ansible creates cluster; Terraform applies workloads; Flux CRs installed from infra/gitops/	Yes (miniature of ground)

Boundary: Terraform provisions (namespaces, PVCs, Deployments, CronJobs); Flux reconciles policy and observability overlays on top. Neither replaces the fleet manifest on the drone.

Developer experience

• Dev Containers define the full toolchain (Python data tooling, DuckDB, compose, linters) — identical on any machine, onboarding in minutes.
• Linux dev VMs are provisioned from the same configuration standard (cloud-init + the provisioning role), so "my VM" and "the CI runner" cannot diverge.
• The virtual swarm (simulator/) is the daily inner loop: change the writer, docker compose up, query the output Parquet, done.