eSlider
57e48e0b55
Problem statement, on-board architecture, Parquet/DuckDB storage design, swarm sync strategy, zero-trust networking, environments, observability, CI/CD delivery with fleet release manifests, roadmap and open questions.
105 lines
5.4 KiB
Markdown
105 lines
5.4 KiB
Markdown
# 11 — CI/CD & delivery
|
|
|
|
How software, models, and configuration reach the fleet — reproducibly, scanned, versioned, and atomically rollback-able. Everything lives inside the air gap.
|
|
|
|
## Source and pipelines: GitLab
|
|
|
|
GitLab (self-managed, on-prem) is the backbone: repositories, CI/CD, and the container registry in one system.
|
|
|
|
- **Shared pipeline templates** — one template library (`ci-templates` repo) defines the standard stages; service repos include and parameterize them instead of copy-pasting YAML.
|
|
- **Multi-arch by default** — every image builds for `linux/amd64` and `linux/arm64` via buildx on dedicated runners; GPU-inference images additionally build against the vendor's L4T-class base for the ARM targets.
|
|
- **Runner fleet on ground k3s** — build runners (amd64 + arm64), a GPU runner for inference smoke tests, and simulation runners that execute virtual-swarm regression scenarios ([06](06-environments.md)).
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph gitlab [GitLab, on-prem]
|
|
SRC["service repos<br/>+ ci-templates"]
|
|
CI["CI pipelines<br/>build · test · scan"]
|
|
REG["GitLab Container Registry<br/>images + generic packages"]
|
|
end
|
|
subgraph quality [Quality gates]
|
|
SQ["SonarQube<br/>code quality"]
|
|
TR["Trivy<br/>image + dependency scan"]
|
|
SIMT["virtual swarm<br/>regression scenarios"]
|
|
end
|
|
subgraph delivery [Delivery]
|
|
MAN["fleet release manifest<br/>semver, digests pinned"]
|
|
MIR["registry mirror<br/>base station"]
|
|
DOCK["dock: verify + apply"]
|
|
end
|
|
SRC --> CI
|
|
CI --> SQ
|
|
CI --> TR
|
|
CI --> SIMT
|
|
CI --> REG
|
|
REG --> MAN
|
|
MAN --> MIR
|
|
MIR --> DOCK
|
|
```
|
|
|
|
## Artifacts: GitLab Registry as the single store
|
|
|
|
One registry for everything, next to the pipelines that produce it:
|
|
|
|
| Artifact | Stored as |
|
|
| --- | --- |
|
|
| Service images (multi-arch) | Container registry, immutable tags + digests |
|
|
| **Detection model weights** (YOLO-like family — architectures and weights vary per mission and improve over iterations) | Generic package registry, semver-versioned, checksummed |
|
|
| Dataset schemas | Generic packages, semver ([03](03-data-platform.md)) |
|
|
| Compose bundles, radio profiles, provisioning configs | Generic packages, semver |
|
|
|
|
Registry hygiene is part of the design: cleanup policies per repository, immutable release tags, access split between CI (write) and mirrors (read).
|
|
|
|
> Consolidating on the GitLab registry removes a separate artifact platform from the stack — one fewer system to run inside the air gap, one auth domain, artifacts adjacent to the pipelines that build them. Scanning moves to Trivy (below).
|
|
|
|
## Quality gates
|
|
|
|
| Gate | Tool | Blocks merge when |
|
|
| --- | --- | --- |
|
|
| Code quality, coverage, static analysis | **SonarQube** | Quality gate red |
|
|
| Image and dependency vulnerabilities | **Trivy** (in CI + scheduled re-scan of released images) | Critical findings without an accepted waiver |
|
|
| Behavioral regression | **Virtual swarm scenarios** — canonical seeds replayed, Parquet outputs asserted (row counts, coverage, staleness budgets) | Any budget exceeded |
|
|
|
|
Scheduled Trivy re-scans matter in an air gap: a released image that was clean in March may carry a known CVE by June; the scan flags it for the next fleet release even though the image never changed.
|
|
|
|
## The fleet release manifest
|
|
|
|
The central delivery idea: **the fleet has exactly one version.**
|
|
|
|
```yaml
|
|
# fleet-release: 3.4.1
|
|
schema_version: 1
|
|
images:
|
|
sensor-ingest: registry.internal/fleet/sensor-ingest@sha256:9f2c… # 2.1.0
|
|
video-analytics: registry.internal/fleet/video-analytics@sha256:5e11… # 3.0.2
|
|
parquet-writer: registry.internal/fleet/parquet-writer@sha256:aa04… # 1.8.0
|
|
state-publisher: registry.internal/fleet/state-publisher@sha256:c7d9… # 1.4.3
|
|
models:
|
|
detector: { package: detector-weights, version: 5.2.0, sha256: "e3b0…" }
|
|
schemas:
|
|
telemetry: 2.1.0
|
|
detections: 1.3.0
|
|
state: 1.2.0
|
|
config:
|
|
compose-bundle: 3.4.0
|
|
radio-profile: production-2
|
|
peer-registry: fleet-42-r7 # provisioning + revocations, see 05
|
|
```
|
|
|
|
- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
|
|
- Images are referenced **by digest**; docks verify signatures and checksums before applying.
|
|
- **Rollback is atomic:** re-apply the previous manifest. No per-service drift, ever — a drone either runs release 3.4.1 in full or 3.4.0 in full.
|
|
- The manifest is what the simulation farm certifies: regression scenarios run against the exact manifest that will ship.
|
|
|
|
## Delivery to the drones
|
|
|
|
1. CI publishes a release manifest; the base-station **registry mirror** pulls all referenced artifacts inside the air gap.
|
|
2. Docked drones fetch the manifest, verify digests/signatures ([05](05-network-security.md)), stage the new Compose bundle, and switch on the next boot cycle.
|
|
3. **Never mid-flight.** Updates are a dock-only operation by construction — the update endpoint does not exist in the flight radio profile.
|
|
|
|
## Developer experience
|
|
|
|
- **Dev Containers** define the full toolchain (Python data tooling, DuckDB, compose, linters) — identical on any machine, onboarding in minutes.
|
|
- **Linux dev VMs** are provisioned from the same configuration standard (cloud-init + the provisioning role), so "my VM" and "the CI runner" cannot diverge.
|
|
- The virtual swarm ([`simulator/`](../simulator/)) is the daily inner loop: change the writer, `docker compose up`, query the output Parquet, done.
|