458 lines
23 KiB
Markdown
458 lines
23 KiB
Markdown
# Development Plan: Control Plane (Admin UI + Observability + Production Ops)
|
|
|
|
## Overview
|
|
|
|
This plan breaks down the Control Plane implementation into milestones ordered by dependency. Each milestone includes:
|
|
- **Tasks** with clear deliverables
|
|
- **Test Requirements** (unit tests + tautological tests + integration tests where applicable)
|
|
- **Dependencies** on previous milestones
|
|
|
|
**Development Approach:**
|
|
1. Complete one milestone at a time
|
|
2. Write tests before implementation (TDD where applicable)
|
|
3. All tests must pass before moving to the next milestone
|
|
4. Mark tasks complete with `[x]` as you progress
|
|
|
|
This plan is intentionally aligned with the style and gating discipline used in sibling repos (see: [gateway/DEVELOPMENT_PLAN.md](file:///Users/vlad/Developer/cloudlysis/gateway/DEVELOPMENT_PLAN.md), [runner/DEVELOPMENT_PLAN.md](file:///Users/vlad/Developer/cloudlysis/runner/DEVELOPMENT_PLAN.md)).
|
|
|
|
---
|
|
|
|
## Milestone 0: Repo Bootstrap (Dev Ergonomics + Guardrails)
|
|
|
|
**Goal:** Establish canonical commands, CI entrypoints, and integration-test gating so later milestones can be executed and verified consistently.
|
|
|
|
### Tasks
|
|
- [x] **0.1** Define canonical local commands for the repo
|
|
- UI:
|
|
- `npm run lint`
|
|
- `npm run typecheck`
|
|
- `npm run test`
|
|
- `npm run build`
|
|
- Control Plane API:
|
|
- `cargo test`
|
|
- `cargo fmt --check`
|
|
- `cargo clippy -- -D warnings`
|
|
- `cargo run -- --help`
|
|
- Docker/Swarm:
|
|
- `docker compose config` validation for local stacks (if used)
|
|
- `docker stack deploy ...` smoke validation for Swarm (gated, see Tests)
|
|
- [x] **0.2** Add a minimal CI workflow that runs the same commands as **0.1**
|
|
- [x] **0.3** Define integration-test gating conventions
|
|
- Docker/Swarm integration tests:
|
|
- Mark as ignored by default and run only when `CONTROL_TEST_DOCKER=1` is set
|
|
- Example: `CONTROL_TEST_DOCKER=1 cargo test -- --ignored`
|
|
- NATS-dependent integration tests:
|
|
- Mark as ignored by default and run only when `CONTROL_TEST_NATS_URL` is set
|
|
- Example: `CONTROL_TEST_NATS_URL=nats://127.0.0.1:4222 cargo test -- --ignored`
|
|
- [x] **0.4** Define baseline operational invariants (checklist for later milestones)
|
|
- No privileged action without RBAC + audit event
|
|
- No multi-step operation without idempotency key + job record
|
|
- Always propagate `tenant_id` (when applicable) end-to-end
|
|
- Always propagate request/flow identifiers end-to-end (logs + downstream calls):
|
|
- `x-request-id` (per HTTP request)
|
|
- `x-correlation-id` (per user-visible flow/job; generated by the Gateway when missing)
|
|
- `traceparent` (W3C trace context; started by the Gateway when missing)
|
|
- Secrets never appear in logs (Authorization headers, tokens, credentials, Grafana admin creds)
|
|
- No tenant-level metrics without bounded cardinality rules
|
|
|
|
### Tests
|
|
- [x] **T0.1** Tautological test: test harness runs for both subprojects (UI + API)
|
|
- [x] **T0.2** Lint + typecheck + unit tests pass
|
|
- [x] **T0.3** Docker config validation passes (compose/stack linting tests)
|
|
|
|
---
|
|
|
|
## Milestone 1: Admin UI Foundation (UltraBase UX Reuse)
|
|
|
|
**Goal:** Bring up the Admin UI with the UltraBase component system and navigation skeleton, adapted to Cloudlysis page structure.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Admin UI builds successfully and passes unit/type checks
|
|
- UI navigation skeleton matches the PRD information architecture
|
|
|
|
### Tasks
|
|
- [x] **1.1** Initialize Admin UI project (Vite + React + TypeScript)
|
|
- Choose and wire lint/typecheck/test/build tooling to match the canonical commands in **0.1**
|
|
- Adopt the baseline dependencies used by UltraBase control-plane admin UI where available
|
|
- Establish UI module layout for: components, pages, routes, API client, auth/session utilities
|
|
- [x] **1.2** Reuse UltraBase UI primitives and styling tokens (adapted, not forked blindly)
|
|
- Buttons, inputs, tables, dropdowns, modal, toast, breadcrumbs
|
|
- [x] **1.3** Implement navigation skeleton and empty pages (route wiring only)
|
|
- Overview
|
|
- Tenants
|
|
- Users
|
|
- Sessions
|
|
- Roles & Permissions
|
|
- Config
|
|
- Definitions
|
|
- Scale & Placement
|
|
- Deployments
|
|
- Observability
|
|
- Audit Log
|
|
- Settings
|
|
- [x] **1.3a** Add correlation-first investigation affordances in the UI skeleton
|
|
- Global search box that accepts `x-request-id`, `x-correlation-id`, or `trace_id`
|
|
- “Investigate” links that open Grafana Explore prefilled for:
|
|
- Loki query scoped to `x-correlation-id` (and `x-request-id` when available)
|
|
- Tempo trace view when a `trace_id` is present
|
|
- Ensure jobs and audit log rows display and copy the relevant ids
|
|
- [x] **1.4** Implement API client stub with consistent error handling and request-id propagation
|
|
- Send `x-request-id` on every request (generate one when missing)
|
|
- Send `x-correlation-id` when continuing an existing UI flow; otherwise omit and use the Gateway-generated value returned in responses
|
|
- Send `traceparent` when continuing an existing trace; otherwise omit and use the Gateway-started trace
|
|
- Echo `x-request-id` and `x-correlation-id` on responses and surface them in error UX
|
|
- Persist the most recent ids in the UI so operators can copy/paste them into support tickets
|
|
|
|
### Tests
|
|
- [x] **T1.1** UI typecheck passes
|
|
- [x] **T1.2** UI build passes
|
|
- [x] **T1.3** Routing smoke test: each route renders without runtime errors (headless DOM test)
|
|
|
|
---
|
|
|
|
## Milestone 2: Control Plane API Foundation (BFF / Admin API)
|
|
|
|
**Goal:** Provide the minimal API surface required for the Admin UI to authenticate, read core state, and display health/metrics.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Control plane API runs as a container and exposes `/health`, `/ready`, `/metrics`
|
|
- Auth integration contract is defined (Gateway as source of truth) and enforced on admin endpoints
|
|
|
|
### Tasks
|
|
- [x] **2.1** Initialize Control Plane API service
|
|
- Rust (Axum + Tokio + tracing) to align with node ecosystem
|
|
- Baseline endpoints: `GET /health`, `GET /ready`, `GET /metrics`
|
|
- [x] **2.2** Add request logging and correlation identifiers
|
|
- `x-request-id` propagation and structured logs (match Gateway conventions)
|
|
- Propagate `x-correlation-id` and `traceparent` on outbound calls
|
|
- Log fields: `request_id`, `correlation_id`, `trace_id`, `principal_id`, `tenant_id` (when applicable)
|
|
- Never log Authorization headers or tokens
|
|
- [x] **2.3** Implement authentication and authorization boundary
|
|
- Validate Gateway-issued access tokens (same signing config as Gateway; Control does not mint tokens)
|
|
- Extract principal identity from token claims (at minimum: `sub`, `session_id`)
|
|
- Enforce permissions at the API boundary (deny-by-default, rights strings stored in Gateway IAM state)
|
|
- Align `x-tenant-id` semantics with Gateway:
|
|
- Tenant-scoped endpoints require `x-tenant-id` and must reject missing/invalid values with 400
|
|
- Platform-scoped endpoints must not depend on `x-tenant-id`
|
|
- Prefer proxying to Gateway for IAM CRUD instead of duplicating identity/RBAC state:
|
|
- Control API may expose a thin BFF surface, but must preserve Gateway status codes and error text for pass-through routes
|
|
- [x] **2.4** Define “job” model for multi-step operations (API contract)
|
|
- `POST /admin/v1/jobs/*` returns `job_id`
|
|
- `GET /admin/v1/jobs/{job_id}` returns status + structured steps + errors
|
|
- Require an idempotency key for job creation (`Idempotency-Key` header), and make repeated creates safe
|
|
|
|
### Tests
|
|
- [x] **T2.1** `GET /health` and `GET /ready` return 200
|
|
- [x] **T2.2** Unauthorized admin calls return 401/403 consistently
|
|
- [x] **T2.3** `x-tenant-id` behavior matches Gateway rules (400 on missing/invalid for tenant-scoped routes)
|
|
- [x] **T2.4** Tautological tests: core state types are Send + Sync
|
|
|
|
---
|
|
|
|
## Milestone 3: Observability Stack Baseline (VM + Loki + Grafana)
|
|
|
|
**Goal:** Include a production-grade observability stack with version-controlled provisioning and Cloudlysis dashboard placeholders wired to existing service metrics.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Grafana starts with provisioned datasources and dashboards
|
|
- vmagent scrapes platform services and VictoriaMetrics can query ingested series
|
|
- Loki is available for log queries (when logs are enabled)
|
|
|
|
### Tasks
|
|
- [x] **3.1** Add observability deployment assets modeled after UltraBase
|
|
- Grafana provisioning for datasources and dashboards
|
|
- vmagent scrape configs for Cloudlysis services + node/Swarm exporters (where applicable)
|
|
- Loki configuration (and optional promtail)
|
|
- [x] **3.1a** Add distributed tracing backend and wiring
|
|
- Tempo (or compatible tracing backend) as a Grafana datasource
|
|
- OTLP receiver path (collector/agent) so platform services can emit traces
|
|
- Grafana Explore is provisioned so operators can jump from logs to traces
|
|
- Require the Gateway to accept and propagate `x-correlation-id` and `traceparent` to upstreams, and to include `correlation_id` and `trace_id` in request spans/log fields
|
|
- [x] **3.2** Implement the base dashboard set from the PRD
|
|
- Operations overview
|
|
- HTTP detail (Gateway route-level)
|
|
- Logs (Loki)
|
|
- Traces (Tempo)
|
|
- Event bus / JetStream
|
|
- Workers (Runner)
|
|
- Storage (libmdbx + node disk)
|
|
- Cluster / Orchestrator
|
|
- [x] **3.3** Add the chosen production-operability dashboards and document required instrumentation
|
|
- Noisy Neighbor & Tenant Health
|
|
- API Regression & Deployment
|
|
- Storage & Event Bus Bottlenecks
|
|
- Infrastructure Exhaustion
|
|
- Standardize build/version labeling across services for correlation (`*_build_info{service,version,git_sha}=1`)
|
|
|
|
### Tests
|
|
- [x] **T3.1** Grafana provisioning files are syntactically valid
|
|
- [x] **T3.2** vmagent config parses and includes all required scrape jobs
|
|
- [x] **T3.3** Tempo (or chosen tracing backend) reaches healthy state in the stack smoke test (gated)
|
|
- [x] **T3.4** Container startup smoke test (compose or Swarm, gated): Grafana + VictoriaMetrics + Loki reach healthy state
|
|
|
|
---
|
|
|
|
## Milestone 4: Tenant + Placement Visibility (Read-Only Ops First)
|
|
|
|
**Goal:** Provide safe, read-only visibility into tenant placement and runtime health across Aggregate/Projection/Runner/Gateway, matching existing placement semantics.
|
|
|
|
### Dependencies
|
|
- Milestone 1 (Admin UI foundation)
|
|
- Milestone 2 (Control Plane API foundation)
|
|
|
|
### Exit Criteria
|
|
- Admin UI can list tenants and show current placement per service kind
|
|
- Placement is sourced from the production control-plane substrate (NATS KV) with a development fallback
|
|
|
|
### Tasks
|
|
- [x] **4.1** Implement placement read APIs
|
|
- Read effective placement from NATS KV (and fallback file for development)
|
|
- Match the Gateway routing config model (placement maps + shard directories + revision semantics)
|
|
- Support per-service-kind placement maps (Aggregate, Projection, Runner) using the same naming conventions used elsewhere (`aggregate_placement`, `projection_placement`, `runner_placement`)
|
|
- [x] **4.2** Implement fleet “health snapshot” APIs
|
|
- Query `/health`, `/ready`, `/metrics` from each service endpoint
|
|
- Normalize into a stable UI response shape
|
|
- [x] **4.3** Implement Admin UI pages:
|
|
- Scale & Placement (read-only)
|
|
- Tenants (read-only with placement summary)
|
|
- Fleet/Topology views (read-only)
|
|
|
|
### Tests
|
|
- [x] **T4.1** Placement config parsing and snapshot endpoints work
|
|
- [x] **T4.2** KV watcher hot-reload swaps placement atomically
|
|
- [x] **T4.3** UI pages render with mocked API responses (component-level tests)
|
|
|
|
---
|
|
|
|
## Milestone 5: Safe Mutations (Drain, Migrate, Reload) via Idempotent Jobs
|
|
|
|
**Goal:** Implement the first high-impact operational workflows with strict guardrails: tenant drain, placement update, and reload.
|
|
|
|
### Dependencies
|
|
- Milestone 4 (read-only ops)
|
|
|
|
### Exit Criteria
|
|
- All operational mutations are executed as jobs with audit events
|
|
- Every mutation supports preflight planning and clear post-conditions
|
|
|
|
### Tasks
|
|
- [x] **5.1** Implement job orchestration primitives in the API
|
|
- step model, retries, cancellation, timeouts
|
|
- per-tenant locking to avoid concurrent conflicting operations
|
|
- [x] **5.2** Implement drain workflow (per service kind where supported)
|
|
- Runner tenant drain semantics (stop acquiring new work, wait for inflight to converge)
|
|
- Aggregate/projection drain semantics via admin endpoints where available
|
|
- Align drain/readiness semantics with the rebalancing contract in [external_prd.md](file:///Users/vlad/Developer/cloudlysis/gateway/external_prd.md)
|
|
- [x] **5.3** Implement migration workflow
|
|
- Plan: drain tenant → update placement → reload routing/config
|
|
- Block unsafe migrations (health/lag/inflight thresholds)
|
|
- [x] **5.4** Implement UI mutation flows
|
|
- modal confirmation + reason required
|
|
- job progress view and audit linkage
|
|
|
|
### Tests
|
|
- [x] **T5.1** Job idempotency: repeated calls with same idempotency key do not duplicate effects
|
|
- [x] **T5.2** Migration plan preflight produces a deterministic action plan
|
|
- [x] **T5.3** Safety gates prevent drain/migrate when invariants fail
|
|
|
|
---
|
|
|
|
## Milestone 6: Deployments + Regression Tooling (Swarm-Aware)
|
|
|
|
**Goal:** Make deployments and regressions observable and controllable from the control plane, with strong “what changed when” correlation.
|
|
|
|
### Dependencies
|
|
- Milestone 3 (observability baseline)
|
|
- Milestone 5 (job orchestration)
|
|
|
|
### Exit Criteria
|
|
- Deployments can be initiated (or at least observed) via the control plane
|
|
- Grafana shows deploy markers; dashboards can compare old vs new versions
|
|
|
|
### Tasks
|
|
- [x] **6.1** Implement Swarm integration (read-only first, then mutations)
|
|
- list services, tasks, images, versions
|
|
- watch update events (start/finish/fail)
|
|
- [x] **6.2** Implement deployment annotations/events
|
|
- write Grafana annotations (or emit a deploy event metric) for vertical markers
|
|
- [x] **6.3** Implement “API Regression & Deployment” dashboard wiring prerequisites
|
|
- enforce build/version labeling (`*_build_info{service,version,git_sha}=1` pattern)
|
|
- ensure scrape relabeling includes `image_tag` where possible
|
|
- [x] **6.4** UI pages
|
|
- Deployments list + detail
|
|
- Per-service “what changed” and “rollback” actions (guarded)
|
|
|
|
### Tests
|
|
- [x] **T6.1** Swarm client abstraction can be mocked and produces deterministic results
|
|
- [x] **T6.2** Annotation writer produces expected Grafana payloads
|
|
- [x] **T6.3** Version labels are present on all services in a metrics snapshot test
|
|
|
|
---
|
|
|
|
## Milestone 7: Full Docker Swarm Deployment (Platform + Observability + Control Plane)
|
|
|
|
**Goal:** Provide a complete Swarm deployment definition for the platform: services in `../` plus the control plane components and the observability stack.
|
|
|
|
### Dependencies
|
|
- Milestone 1 (Admin UI foundation)
|
|
- Milestone 2 (Control Plane API foundation)
|
|
- Milestone 3 (Observability baseline)
|
|
- Milestone 5 (safe mutations baseline)
|
|
|
|
### Exit Criteria
|
|
- `docker stack deploy` brings up:
|
|
- Gateway + Aggregate + Projection + Runner (from `../`)
|
|
- Control Plane API + Admin UI
|
|
- VictoriaMetrics + vmagent + Grafana + Loki (+ optional promtail)
|
|
- All services are reachable via overlay networks and pass health checks
|
|
- Smoke and integration tests pass end-to-end (gated, but required before milestone completion)
|
|
|
|
### Tasks
|
|
- [x] **7.1** Define Swarm networks, secrets, and configs
|
|
- overlay network segmentation (public vs internal)
|
|
- secrets for auth/signing keys, NATS credentials (if used), Grafana admin creds (or provisioning)
|
|
- [x] **7.2** Define Swarm stack files
|
|
- base platform stack (gateway/aggregate/projection/runner)
|
|
- control plane stack (api + ui)
|
|
- observability stack (vm/vmagent/grafana/loki/promtail)
|
|
- [x] **7.3** Define placement constraints and scaling defaults
|
|
- node labels for tenant ranges and infrastructure roles
|
|
- replica defaults and update policies
|
|
- [x] **7.4** Define deployment verification and rollback playbooks (as executable checks)
|
|
- post-deploy checks: `/health`, `/ready`, `/metrics`, dashboard provisioning
|
|
- rollbacks: service update rollback hooks and job safety checks
|
|
|
|
### Tests
|
|
- [x] **T7.1** Stack YAML parses and validates (unit test)
|
|
- [x] **T7.2** Swarm smoke test (requires `CONTROL_TEST_DOCKER=1`)
|
|
- deploy stacks
|
|
- wait for healthy state
|
|
- verify Grafana dashboards provisioned and VictoriaMetrics receives samples
|
|
- [x] **T7.3** End-to-end “control plane can see the fleet” test (requires docker)
|
|
- UI/API can query placement + health snapshots for all services
|
|
|
|
---
|
|
|
|
## Milestone 8: Config Registry + Safe Change Management (Plan/Apply/Rollback)
|
|
|
|
**Goal:** Make configuration first-class, versioned, validated, and safely mutable from the control plane, while keeping production and development sources consistent.
|
|
|
|
### Dependencies
|
|
- Milestone 2 (Control Plane API foundation)
|
|
- Milestone 5 (safe mutations baseline)
|
|
- Milestone 7 (Swarm deployment baseline)
|
|
|
|
### Exit Criteria
|
|
- Operators can list, view, validate, and safely apply config changes with audit + idempotent jobs
|
|
- Config changes have revision semantics and are roll-backable
|
|
- Gatekeeper safety checks prevent applying invalid or unsafe configs
|
|
|
|
### Tasks
|
|
- [x] **8.1** Inventory and classify configuration surfaces (platform-wide)
|
|
- classify as: static boot config (env/secrets), dynamic runtime config (KV), large immutable artifacts (S3/docs)
|
|
- map current sources per domain:
|
|
- Gateway routing config (`config/routing/dev.json` / production KV)
|
|
- Placement config (`config/placement/dev.json` / production KV)
|
|
- Runner definitions (effects/sagas) (documents/S3) and activation config (KV)
|
|
- Observability provisioning (Swarm configs + repo-managed assets)
|
|
- Control plane feature flags (KV)
|
|
- [~] **8.2** Define a Config Registry contract in the Control API
|
|
- **Implemented (initial)**:
|
|
- config identity: `{domain}` (routing|placement)
|
|
- metadata: `revision` (KV revision when using NATS), and `source` info (file vs nats)
|
|
- storage policy per config: `source=dev_file | nats_kv`
|
|
- **Still needed**:
|
|
- `{domain, name, scope}` and richer metadata (`updated_at`, `updated_by`, `sha256`)
|
|
- history API for KV-backed configs
|
|
- [x] **8.3** Implement config storage abstraction (dev + prod)
|
|
- dev: file-backed, atomic write (tmp + rename), hot-reload where applicable
|
|
- prod: NATS KV for dynamic configs (revisioned values + watch streams)
|
|
- consistent error model: decode/validate/source errors are distinguishable and safe
|
|
- [x] **8.4** Add read-only config APIs
|
|
- `GET /admin/v1/config` list domains
|
|
- `GET /admin/v1/config/{domain}` fetch current value + revision + source
|
|
- (history not implemented yet)
|
|
- [~] **8.5** Add validate/plan/apply/rollback mutation workflows as jobs
|
|
- **Implemented**:
|
|
- `POST /admin/v1/jobs/config/validate` (job, idempotency key required)
|
|
- `POST /admin/v1/jobs/config/apply` (job, idempotency key required, backup + apply)
|
|
- `POST /admin/v1/jobs/config/rollback` (job, idempotency key required, restore last backup)
|
|
- per-domain locking to avoid concurrent config mutations
|
|
- **Still needed**:
|
|
- `POST /admin/v1/plan/config/apply` deterministic plan (diff + impacted services)
|
|
- richer post-conditions (routing resolution sampling, fleet consistency checks, etc.)
|
|
- [~] **8.6** Implement initial config domains end-to-end
|
|
- **Gateway routing config**:
|
|
- implemented: schema validation via JSON decode
|
|
- still needed: semantic validation (tenant entries/shard directories/endpoints URL parsing) + sampled routing verification
|
|
- **Placement config**:
|
|
- implemented: schema validation via JSON decode
|
|
- still needed: semantic validation (targets non-empty, etc.) + fleet snapshot consistency checks
|
|
- [x] **8.7** Implement Admin UI “Config” page for safe operations
|
|
- list + view configs with revision/sha/audit linkage
|
|
- editor for JSON (and YAML when supported by the domain)
|
|
- validate button (server-side) and apply/rollback flows as jobs with reason required
|
|
|
|
### Tests
|
|
- [x] **T8.1** Unit tests: config decode/encode stability for each config domain
|
|
- routing/placement decode is enforced by server-side validate job (schema-level)
|
|
- [ ] **T8.2** Unit tests: validation rejects unsafe configs with stable error codes/messages
|
|
- [ ] **T8.3** Unit tests: plan generation is deterministic for same inputs
|
|
- [x] **T8.4** Integration tests (env-gated):
|
|
- NATS KV config apply + rollback via Control API (requires `CONTROL_TEST_NATS=1` + `CONTROL_TEST_NATS_URL`)
|
|
- (Gateway route-resolution E2E verification still pending)
|
|
- [x] **T8.5** UI tests: config page renders, validate/apply/rollback flows navigate to job progress
|
|
|
|
---
|
|
|
|
## Milestone 9: Control Node Management (Inventory, Drift, and Safer Ops)
|
|
|
|
**Goal:** Improve how the control plane understands and manages the live control node and platform state: node inventory, config drift detection, and safer operational guardrails.
|
|
|
|
### Dependencies
|
|
- Milestone 7 (Swarm deployment baseline)
|
|
- Milestone 8 (config registry + safe change management)
|
|
|
|
### Exit Criteria
|
|
- Control plane provides a reliable “what is running vs what should be running” view
|
|
- Config drift is detectable and actionable
|
|
- Core operational actions are guarded by preflight checks and produce audit trails
|
|
|
|
### Tasks
|
|
- [x] **9.1** Define a “desired vs observed” model for platform state
|
|
- desired: Swarm stacks + config registry revisions
|
|
- observed: live service/task state + effective runtime configs
|
|
- drift categories: missing, extra, version mismatch, config mismatch, unhealthy
|
|
- [~] **9.2** Improve Swarm observation fidelity
|
|
- implemented (initial): docker-cli-backed Swarm observation (`CONTROL_SWARM_MODE=docker`)
|
|
- still needed: direct Docker API client (avoid shelling out), richer normalization, and wiring into production stacks
|
|
- keep file source as a dev fallback for deterministic tests
|
|
- normalize service identity: `{service, image_tag, git_sha, updated_at}`
|
|
- [x] **9.3** Add drift APIs and UI views
|
|
- `GET /admin/v1/platform/drift` returns drift summary + actionable items
|
|
- UI: “Platform Drift” page with filters and links to remediate jobs
|
|
- [ ] **9.4** Add safer operational guardrails as reusable checks
|
|
- preflight checks for:
|
|
- service unhealthy / crashloop
|
|
- tenant migration safety thresholds (lag/inflight)
|
|
- config apply safety (impact radius, sampled verify)
|
|
- consistent failure modes: clear reason + audit entry, no partial side effects
|
|
- [ ] **9.5** Add operational playbooks as executable checks
|
|
- post-deploy verification suite callable as an idempotent job
|
|
- rollback verification suite callable as an idempotent job
|
|
|
|
### Tests
|
|
- [x] **T9.1** Unit tests: drift classification for synthetic desired/observed fixtures
|
|
- [x] **T9.2** Integration tests (docker-gated): drift view detects intentional mismatches in a local Swarm
|
|
- requires `CONTROL_TEST_DOCKER=1` and an active local Swarm node
|
|
- [x] **T9.3** UI tests: drift page renders in route smoke test
|