342 lines
16 KiB
Markdown
342 lines
16 KiB
Markdown
# Development Plan: Control Plane (Admin UI + Observability + Production Ops)
|
|
|
|
## Overview
|
|
|
|
This plan breaks down the Control Plane implementation into milestones ordered by dependency. Each milestone includes:
|
|
- **Tasks** with clear deliverables
|
|
- **Test Requirements** (unit tests + tautological tests + integration tests where applicable)
|
|
- **Dependencies** on previous milestones
|
|
|
|
**Development Approach:**
|
|
1. Complete one milestone at a time
|
|
2. Write tests before implementation (TDD where applicable)
|
|
3. All tests must pass before moving to the next milestone
|
|
4. Mark tasks complete with `[x]` as you progress
|
|
|
|
This plan is intentionally aligned with the style and gating discipline used in sibling repos (see: [gateway/DEVELOPMENT_PLAN.md](file:///Users/vlad/Developer/cloudlysis/gateway/DEVELOPMENT_PLAN.md), [runner/DEVELOPMENT_PLAN.md](file:///Users/vlad/Developer/cloudlysis/runner/DEVELOPMENT_PLAN.md)).
|
|
|
|
---
|
|
|
|
## Milestone 0: Repo Bootstrap (Dev Ergonomics + Guardrails)
|
|
|
|
**Goal:** Establish canonical commands, CI entrypoints, and integration-test gating so later milestones can be executed and verified consistently.
|
|
|
|
### Tasks
|
|
- [x] **0.1** Define canonical local commands for the repo
|
|
- UI:
|
|
- `npm run lint`
|
|
- `npm run typecheck`
|
|
- `npm run test`
|
|
- `npm run build`
|
|
- Control Plane API:
|
|
- `cargo test`
|
|
- `cargo fmt --check`
|
|
- `cargo clippy -- -D warnings`
|
|
- `cargo run -- --help`
|
|
- Docker/Swarm:
|
|
- `docker compose config` validation for local stacks (if used)
|
|
- `docker stack deploy ...` smoke validation for Swarm (gated, see Tests)
|
|
- [x] **0.2** Add a minimal CI workflow that runs the same commands as **0.1**
|
|
- [x] **0.3** Define integration-test gating conventions
|
|
- Docker/Swarm integration tests:
|
|
- Mark as ignored by default and run only when `CONTROL_TEST_DOCKER=1` is set
|
|
- Example: `CONTROL_TEST_DOCKER=1 cargo test -- --ignored`
|
|
- NATS-dependent integration tests:
|
|
- Mark as ignored by default and run only when `CONTROL_TEST_NATS_URL` is set
|
|
- Example: `CONTROL_TEST_NATS_URL=nats://127.0.0.1:4222 cargo test -- --ignored`
|
|
- [x] **0.4** Define baseline operational invariants (checklist for later milestones)
|
|
- No privileged action without RBAC + audit event
|
|
- No multi-step operation without idempotency key + job record
|
|
- Always propagate `tenant_id` (when applicable) end-to-end
|
|
- Always propagate request/flow identifiers end-to-end (logs + downstream calls):
|
|
- `x-request-id` (per HTTP request)
|
|
- `x-correlation-id` (per user-visible flow/job; generated by the Gateway when missing)
|
|
- `traceparent` (W3C trace context; started by the Gateway when missing)
|
|
- Secrets never appear in logs (Authorization headers, tokens, credentials, Grafana admin creds)
|
|
- No tenant-level metrics without bounded cardinality rules
|
|
|
|
### Tests
|
|
- [x] **T0.1** Tautological test: test harness runs for both subprojects (UI + API)
|
|
- [x] **T0.2** Lint + typecheck + unit tests pass
|
|
- [x] **T0.3** Docker config validation passes (compose/stack linting tests)
|
|
|
|
---
|
|
|
|
## Milestone 1: Admin UI Foundation (UltraBase UX Reuse)
|
|
|
|
**Goal:** Bring up the Admin UI with the UltraBase component system and navigation skeleton, adapted to Cloudlysis page structure.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Admin UI builds successfully and passes unit/type checks
|
|
- UI navigation skeleton matches the PRD information architecture
|
|
|
|
### Tasks
|
|
- [x] **1.1** Initialize Admin UI project (Vite + React + TypeScript)
|
|
- Choose and wire lint/typecheck/test/build tooling to match the canonical commands in **0.1**
|
|
- Adopt the baseline dependencies used by UltraBase control-plane admin UI where available
|
|
- Establish UI module layout for: components, pages, routes, API client, auth/session utilities
|
|
- [x] **1.2** Reuse UltraBase UI primitives and styling tokens (adapted, not forked blindly)
|
|
- Buttons, inputs, tables, dropdowns, modal, toast, breadcrumbs
|
|
- [x] **1.3** Implement navigation skeleton and empty pages (route wiring only)
|
|
- Overview
|
|
- Tenants
|
|
- Users
|
|
- Sessions
|
|
- Roles & Permissions
|
|
- Config
|
|
- Definitions
|
|
- Scale & Placement
|
|
- Deployments
|
|
- Observability
|
|
- Audit Log
|
|
- Settings
|
|
- [x] **1.3a** Add correlation-first investigation affordances in the UI skeleton
|
|
- Global search box that accepts `x-request-id`, `x-correlation-id`, or `trace_id`
|
|
- “Investigate” links that open Grafana Explore prefilled for:
|
|
- Loki query scoped to `x-correlation-id` (and `x-request-id` when available)
|
|
- Tempo trace view when a `trace_id` is present
|
|
- Ensure jobs and audit log rows display and copy the relevant ids
|
|
- [x] **1.4** Implement API client stub with consistent error handling and request-id propagation
|
|
- Send `x-request-id` on every request (generate one when missing)
|
|
- Send `x-correlation-id` when continuing an existing UI flow; otherwise omit and use the Gateway-generated value returned in responses
|
|
- Send `traceparent` when continuing an existing trace; otherwise omit and use the Gateway-started trace
|
|
- Echo `x-request-id` and `x-correlation-id` on responses and surface them in error UX
|
|
- Persist the most recent ids in the UI so operators can copy/paste them into support tickets
|
|
|
|
### Tests
|
|
- [x] **T1.1** UI typecheck passes
|
|
- [x] **T1.2** UI build passes
|
|
- [x] **T1.3** Routing smoke test: each route renders without runtime errors (headless DOM test)
|
|
|
|
---
|
|
|
|
## Milestone 2: Control Plane API Foundation (BFF / Admin API)
|
|
|
|
**Goal:** Provide the minimal API surface required for the Admin UI to authenticate, read core state, and display health/metrics.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Control plane API runs as a container and exposes `/health`, `/ready`, `/metrics`
|
|
- Auth integration contract is defined (Gateway as source of truth) and enforced on admin endpoints
|
|
|
|
### Tasks
|
|
- [x] **2.1** Initialize Control Plane API service
|
|
- Rust (Axum + Tokio + tracing) to align with node ecosystem
|
|
- Baseline endpoints: `GET /health`, `GET /ready`, `GET /metrics`
|
|
- [x] **2.2** Add request logging and correlation identifiers
|
|
- `x-request-id` propagation and structured logs (match Gateway conventions)
|
|
- Propagate `x-correlation-id` and `traceparent` on outbound calls
|
|
- Log fields: `request_id`, `correlation_id`, `trace_id`, `principal_id`, `tenant_id` (when applicable)
|
|
- Never log Authorization headers or tokens
|
|
- [x] **2.3** Implement authentication and authorization boundary
|
|
- Validate Gateway-issued access tokens (same signing config as Gateway; Control does not mint tokens)
|
|
- Extract principal identity from token claims (at minimum: `sub`, `session_id`)
|
|
- Enforce permissions at the API boundary (deny-by-default, rights strings stored in Gateway IAM state)
|
|
- Align `x-tenant-id` semantics with Gateway:
|
|
- Tenant-scoped endpoints require `x-tenant-id` and must reject missing/invalid values with 400
|
|
- Platform-scoped endpoints must not depend on `x-tenant-id`
|
|
- Prefer proxying to Gateway for IAM CRUD instead of duplicating identity/RBAC state:
|
|
- Control API may expose a thin BFF surface, but must preserve Gateway status codes and error text for pass-through routes
|
|
- [x] **2.4** Define “job” model for multi-step operations (API contract)
|
|
- `POST /admin/v1/jobs/*` returns `job_id`
|
|
- `GET /admin/v1/jobs/{job_id}` returns status + structured steps + errors
|
|
- Require an idempotency key for job creation (`Idempotency-Key` header), and make repeated creates safe
|
|
|
|
### Tests
|
|
- [x] **T2.1** `GET /health` and `GET /ready` return 200
|
|
- [x] **T2.2** Unauthorized admin calls return 401/403 consistently
|
|
- [x] **T2.3** `x-tenant-id` behavior matches Gateway rules (400 on missing/invalid for tenant-scoped routes)
|
|
- [x] **T2.4** Tautological tests: core state types are Send + Sync
|
|
|
|
---
|
|
|
|
## Milestone 3: Observability Stack Baseline (VM + Loki + Grafana)
|
|
|
|
**Goal:** Include a production-grade observability stack with version-controlled provisioning and Cloudlysis dashboard placeholders wired to existing service metrics.
|
|
|
|
### Dependencies
|
|
- Milestone 0 (repo bootstrap)
|
|
|
|
### Exit Criteria
|
|
- Grafana starts with provisioned datasources and dashboards
|
|
- vmagent scrapes platform services and VictoriaMetrics can query ingested series
|
|
- Loki is available for log queries (when logs are enabled)
|
|
|
|
### Tasks
|
|
- [x] **3.1** Add observability deployment assets modeled after UltraBase
|
|
- Grafana provisioning for datasources and dashboards
|
|
- vmagent scrape configs for Cloudlysis services + node/Swarm exporters (where applicable)
|
|
- Loki configuration (and optional promtail)
|
|
- [x] **3.1a** Add distributed tracing backend and wiring
|
|
- Tempo (or compatible tracing backend) as a Grafana datasource
|
|
- OTLP receiver path (collector/agent) so platform services can emit traces
|
|
- Grafana Explore is provisioned so operators can jump from logs to traces
|
|
- Require the Gateway to accept and propagate `x-correlation-id` and `traceparent` to upstreams, and to include `correlation_id` and `trace_id` in request spans/log fields
|
|
- [x] **3.2** Implement the base dashboard set from the PRD
|
|
- Operations overview
|
|
- HTTP detail (Gateway route-level)
|
|
- Logs (Loki)
|
|
- Traces (Tempo)
|
|
- Event bus / JetStream
|
|
- Workers (Runner)
|
|
- Storage (libmdbx + node disk)
|
|
- Cluster / Orchestrator
|
|
- [x] **3.3** Add the chosen production-operability dashboards and document required instrumentation
|
|
- Noisy Neighbor & Tenant Health
|
|
- API Regression & Deployment
|
|
- Storage & Event Bus Bottlenecks
|
|
- Infrastructure Exhaustion
|
|
- Standardize build/version labeling across services for correlation (`*_build_info{service,version,git_sha}=1`)
|
|
|
|
### Tests
|
|
- [x] **T3.1** Grafana provisioning files are syntactically valid
|
|
- [x] **T3.2** vmagent config parses and includes all required scrape jobs
|
|
- [x] **T3.3** Tempo (or chosen tracing backend) reaches healthy state in the stack smoke test (gated)
|
|
- [x] **T3.4** Container startup smoke test (compose or Swarm, gated): Grafana + VictoriaMetrics + Loki reach healthy state
|
|
|
|
---
|
|
|
|
## Milestone 4: Tenant + Placement Visibility (Read-Only Ops First)
|
|
|
|
**Goal:** Provide safe, read-only visibility into tenant placement and runtime health across Aggregate/Projection/Runner/Gateway, matching existing placement semantics.
|
|
|
|
### Dependencies
|
|
- Milestone 1 (Admin UI foundation)
|
|
- Milestone 2 (Control Plane API foundation)
|
|
|
|
### Exit Criteria
|
|
- Admin UI can list tenants and show current placement per service kind
|
|
- Placement is sourced from the production control-plane substrate (NATS KV) with a development fallback
|
|
|
|
### Tasks
|
|
- [x] **4.1** Implement placement read APIs
|
|
- Read effective placement from NATS KV (and fallback file for development)
|
|
- Match the Gateway routing config model (placement maps + shard directories + revision semantics)
|
|
- Support per-service-kind placement maps (Aggregate, Projection, Runner) using the same naming conventions used elsewhere (`aggregate_placement`, `projection_placement`, `runner_placement`)
|
|
- [x] **4.2** Implement fleet “health snapshot” APIs
|
|
- Query `/health`, `/ready`, `/metrics` from each service endpoint
|
|
- Normalize into a stable UI response shape
|
|
- [x] **4.3** Implement Admin UI pages:
|
|
- Scale & Placement (read-only)
|
|
- Tenants (read-only with placement summary)
|
|
- Fleet/Topology views (read-only)
|
|
|
|
### Tests
|
|
- [x] **T4.1** Placement config parsing and snapshot endpoints work
|
|
- [x] **T4.2** KV watcher hot-reload swaps placement atomically
|
|
- [x] **T4.3** UI pages render with mocked API responses (component-level tests)
|
|
|
|
---
|
|
|
|
## Milestone 5: Safe Mutations (Drain, Migrate, Reload) via Idempotent Jobs
|
|
|
|
**Goal:** Implement the first high-impact operational workflows with strict guardrails: tenant drain, placement update, and reload.
|
|
|
|
### Dependencies
|
|
- Milestone 4 (read-only ops)
|
|
|
|
### Exit Criteria
|
|
- All operational mutations are executed as jobs with audit events
|
|
- Every mutation supports preflight planning and clear post-conditions
|
|
|
|
### Tasks
|
|
- [x] **5.1** Implement job orchestration primitives in the API
|
|
- step model, retries, cancellation, timeouts
|
|
- per-tenant locking to avoid concurrent conflicting operations
|
|
- [x] **5.2** Implement drain workflow (per service kind where supported)
|
|
- Runner tenant drain semantics (stop acquiring new work, wait for inflight to converge)
|
|
- Aggregate/projection drain semantics via admin endpoints where available
|
|
- Align drain/readiness semantics with the rebalancing contract in [external_prd.md](file:///Users/vlad/Developer/cloudlysis/gateway/external_prd.md)
|
|
- [x] **5.3** Implement migration workflow
|
|
- Plan: drain tenant → update placement → reload routing/config
|
|
- Block unsafe migrations (health/lag/inflight thresholds)
|
|
- [x] **5.4** Implement UI mutation flows
|
|
- modal confirmation + reason required
|
|
- job progress view and audit linkage
|
|
|
|
### Tests
|
|
- [x] **T5.1** Job idempotency: repeated calls with same idempotency key do not duplicate effects
|
|
- [x] **T5.2** Migration plan preflight produces a deterministic action plan
|
|
- [x] **T5.3** Safety gates prevent drain/migrate when invariants fail
|
|
|
|
---
|
|
|
|
## Milestone 6: Deployments + Regression Tooling (Swarm-Aware)
|
|
|
|
**Goal:** Make deployments and regressions observable and controllable from the control plane, with strong “what changed when” correlation.
|
|
|
|
### Dependencies
|
|
- Milestone 3 (observability baseline)
|
|
- Milestone 5 (job orchestration)
|
|
|
|
### Exit Criteria
|
|
- Deployments can be initiated (or at least observed) via the control plane
|
|
- Grafana shows deploy markers; dashboards can compare old vs new versions
|
|
|
|
### Tasks
|
|
- [x] **6.1** Implement Swarm integration (read-only first, then mutations)
|
|
- list services, tasks, images, versions
|
|
- watch update events (start/finish/fail)
|
|
- [x] **6.2** Implement deployment annotations/events
|
|
- write Grafana annotations (or emit a deploy event metric) for vertical markers
|
|
- [x] **6.3** Implement “API Regression & Deployment” dashboard wiring prerequisites
|
|
- enforce build/version labeling (`*_build_info{service,version,git_sha}=1` pattern)
|
|
- ensure scrape relabeling includes `image_tag` where possible
|
|
- [x] **6.4** UI pages
|
|
- Deployments list + detail
|
|
- Per-service “what changed” and “rollback” actions (guarded)
|
|
|
|
### Tests
|
|
- [x] **T6.1** Swarm client abstraction can be mocked and produces deterministic results
|
|
- [x] **T6.2** Annotation writer produces expected Grafana payloads
|
|
- [x] **T6.3** Version labels are present on all services in a metrics snapshot test
|
|
|
|
---
|
|
|
|
## Milestone 7: Full Docker Swarm Deployment (Platform + Observability + Control Plane)
|
|
|
|
**Goal:** Provide a complete Swarm deployment definition for the platform: services in `../` plus the control plane components and the observability stack.
|
|
|
|
### Dependencies
|
|
- Milestone 1 (Admin UI foundation)
|
|
- Milestone 2 (Control Plane API foundation)
|
|
- Milestone 3 (Observability baseline)
|
|
- Milestone 5 (safe mutations baseline)
|
|
|
|
### Exit Criteria
|
|
- `docker stack deploy` brings up:
|
|
- Gateway + Aggregate + Projection + Runner (from `../`)
|
|
- Control Plane API + Admin UI
|
|
- VictoriaMetrics + vmagent + Grafana + Loki (+ optional promtail)
|
|
- All services are reachable via overlay networks and pass health checks
|
|
- Smoke and integration tests pass end-to-end (gated, but required before milestone completion)
|
|
|
|
### Tasks
|
|
- [x] **7.1** Define Swarm networks, secrets, and configs
|
|
- overlay network segmentation (public vs internal)
|
|
- secrets for auth/signing keys, NATS credentials (if used), Grafana admin creds (or provisioning)
|
|
- [x] **7.2** Define Swarm stack files
|
|
- base platform stack (gateway/aggregate/projection/runner)
|
|
- control plane stack (api + ui)
|
|
- observability stack (vm/vmagent/grafana/loki/promtail)
|
|
- [x] **7.3** Define placement constraints and scaling defaults
|
|
- node labels for tenant ranges and infrastructure roles
|
|
- replica defaults and update policies
|
|
- [x] **7.4** Define deployment verification and rollback playbooks (as executable checks)
|
|
- post-deploy checks: `/health`, `/ready`, `/metrics`, dashboard provisioning
|
|
- rollbacks: service update rollback hooks and job safety checks
|
|
|
|
### Tests
|
|
- [x] **T7.1** Stack YAML parses and validates (unit test)
|
|
- [x] **T7.2** Swarm smoke test (requires `CONTROL_TEST_DOCKER=1`)
|
|
- deploy stacks
|
|
- wait for healthy state
|
|
- verify Grafana dashboards provisioned and VictoriaMetrics receives samples
|
|
- [x] **T7.3** End-to-end “control plane can see the fleet” test (requires docker)
|
|
- UI/API can query placement + health snapshots for all services
|