swarm-house/docs/11-cicd-delivery.md

# 11 — CI/CD & delivery

How software, models, and configuration reach the fleet — reproducibly, scanned, versioned, and atomically rollback-able. Everything lives inside the air gap.

> **Decisions:** [ADR-0006 — IaC boundaries & fleet manifest](adr/ADR-0006-iac-boundaries.md), [ADR-0007 — non-root image](adr/ADR-0007-non-root-image.md).

## Source and pipelines: GitLab

GitLab (self-managed, on-prem) is the backbone: repositories, CI/CD, and the container registry in one system.

- **Shared pipeline templates** — one template library (`ci-templates` repo) defines the standard stages; service repos include and parameterize them instead of copy-pasting YAML.
- **Multi-arch by default** — every image builds for `linux/amd64` and `linux/arm64` via buildx on dedicated runners; GPU-inference images additionally build against the vendor's L4T-class base for the ARM targets.
- **Runner fleet on ground k3s** — build runners (amd64 + arm64), a GPU runner for inference smoke tests, and simulation runners that execute virtual-swarm regression scenarios ([06](06-environments.md)).

```mermaid
graph LR
    subgraph gitlab [GitLab, on-prem]
        SRC["service repos<br/>+ ci-templates"]
        CI["CI pipelines<br/>build · test · scan"]
        REG["GitLab Container Registry<br/>images + generic packages"]
    end
    subgraph quality [Quality gates]
        SQ["SonarQube<br/>code quality"]
        TR["Trivy<br/>image + dependency scan"]
        SIMT["virtual swarm<br/>regression scenarios"]
    end
    subgraph delivery [Delivery]
        MAN["fleet release manifest<br/>semver, digests pinned"]
        MIR["registry mirror<br/>base station"]
        DOCK["dock: verify + apply"]
    end
    SRC --> CI
    CI --> SQ
    CI --> TR
    CI --> SIMT
    CI --> REG
    REG --> MAN
    MAN --> MIR
    MIR --> DOCK
```

## Artifacts: GitLab Registry as the single store

One registry for everything, next to the pipelines that produce it:

| Artifact | Stored as |
| --- | --- |
| Service images (multi-arch) | Container registry, immutable tags + digests |
| **Detection model weights** (YOLO-like family — architectures and weights vary per mission and improve over iterations) | Generic package registry, semver-versioned, checksummed |
| Dataset schemas | Generic packages, semver ([03](03-data-platform.md)) |
| Compose bundles, radio profiles, provisioning configs | Generic packages, semver |

Registry hygiene is part of the design: cleanup policies per repository, immutable release tags, access split between CI (write) and mirrors (read).

> Consolidating on the GitLab registry removes a separate artifact platform from the stack — one fewer system to run inside the air gap, one auth domain, artifacts adjacent to the pipelines that build them. Scanning moves to Trivy (below).

## Quality gates

| Gate | Tool | Blocks merge when |
| --- | --- | --- |
| Code quality, coverage, static analysis | **SonarQube** | Quality gate red |
| Image and dependency vulnerabilities | **Trivy** (in CI + scheduled re-scan of released images) | Critical findings without an accepted waiver |
| Behavioral regression | **Virtual swarm scenarios** — canonical seeds replayed, Parquet outputs asserted (row counts, coverage, staleness budgets) | Any budget exceeded |

Scheduled Trivy re-scans matter in an air gap: a released image that was clean in March may carry a known CVE by June; the scan flags it for the next fleet release even though the image never changed.

## The fleet release manifest

The central delivery idea: **the fleet has exactly one version.**

```yaml
# fleet-release: 3.4.1
schema_version: 1
images:
  sensor-ingest:    registry.internal/fleet/sensor-ingest@sha256:9f2c…   # 2.1.0
  video-analytics:  registry.internal/fleet/video-analytics@sha256:5e11… # 3.0.2
  parquet-writer:   registry.internal/fleet/parquet-writer@sha256:aa04…  # 1.8.0
  state-publisher:  registry.internal/fleet/state-publisher@sha256:c7d9… # 1.4.3
models:
  detector: { package: detector-weights, version: 5.2.0, sha256: "e3b0…" }
schemas:
  telemetry: 2.1.0
  detections: 1.3.0
  state: 1.2.0
config:
  compose-bundle: 3.4.0
  radio-profile: production-2
  peer-registry: fleet-42-r7      # provisioning + revocations, see [05 — Network & security](05-network-security.md)
```

- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
- Images are referenced **by digest**; docks verify signatures and checksums before applying.
- **Rollback is atomic:** re-apply the previous manifest. No per-service drift, ever — a drone either runs release 3.4.1 in full or 3.4.0 in full.
- The manifest is what the simulation farm certifies: regression scenarios run against the exact manifest that will ship.

## Delivery to the drones

1. CI publishes a release manifest; the base-station **registry mirror** pulls all referenced artifacts inside the air gap.
2. Docked drones fetch the manifest, verify digests/signatures ([05](05-network-security.md)), stage the new Compose bundle, and switch on the next boot cycle.
3. **Never mid-flight.** Updates are a dock-only operation by construction — the update endpoint does not exist in the flight radio profile.

## GitOps on the ground segment

The fleet release manifest handles **drones** (atomic, digest-pinned, dock-only). The **ground k3s segment** uses a different tool: [Flux](https://fluxcd.io/) reconciles long-lived configuration from git — dashboard bundles, Prometheus rules, offload schedules, policy labels — without reprovisioning PVCs or re-running full Terraform applies for every label change.

```mermaid
graph LR
    subgraph git [Git, on-prem]
        REPO["service repos"]
        GITOPS["infra/gitops/<br/>ground overlay"]
    end
    subgraph ground_k3s [Ground k3s]
        FLUX["Flux controllers"]
        WH["warehouse workloads<br/>(Terraform-provisioned)"]
        CFG["policy ConfigMaps<br/>dashboard bundles"]
    end
    subgraph drones [Fleet]
        MAN["fleet release manifest"]
        DOCK["dock: verify + apply"]
    end
    REPO --> CI
    GITOPS --> FLUX
    FLUX --> CFG
    MAN --> DOCK
```

| Segment | Delivery mechanism | Reconciler in production? |
| --- | --- | --- |
| Drones | Fleet release manifest via registry mirror | **No** — no API server mid-flight; dock applies once |
| Ground k3s (warehouse, dashboards, CI runners) | Terraform for shape + **Flux** for ongoing config | **Yes** — stationary cluster, wired network, git is the source of truth |
| Dev simulation (k3d) | Ansible creates cluster; Terraform applies workloads; Flux CRs installed from [`infra/gitops/`](../infra/gitops/) | Yes (miniature of ground) |

Boundary: **Terraform provisions** (namespaces, PVCs, Deployments, CronJobs); **Flux reconciles** policy and observability overlays on top. Neither replaces the fleet manifest on the drone.

## Developer experience

- **Dev Containers** define the full toolchain (Python data tooling, DuckDB, compose, linters) — identical on any machine, onboarding in minutes.
- **Linux dev VMs** are provisioned from the same configuration standard (cloud-init + the provisioning role), so "my VM" and "the CI runner" cannot diverge.
- The virtual swarm ([`simulator/`](../simulator/)) is the daily inner loop: change the writer, `docker compose up`, query the output Parquet, done.