### 🧱 Component: Control Plane (Admin UI + Monitoring + Production Ops)

**Definition:**  
This repository hosts the **platform control plane**:
1) the **Admin UI** used by platform operators and admins to manage users/roles/sessions, tenants, configuration, definitions, and production scaling; and  
2) the **observability stack** and **production dashboards** (VictoriaMetrics + Loki + Grafana, plus alerting/scrape config) required to operate the platform in production.

The control plane is the “single pane of glass” and the “safe hands” layer: it does not replace node runtime logic; it coordinates existing node capabilities and exposes them with strict RBAC, auditability, and operational guardrails.

---

## **Context: Existing Node Repositories (../)**

This PRD is derived from the currently implemented node repos in `../`:
- **Aggregate**: expects a control node to manage tenant placement and scaling operations, including tenant migrations ([aggregate/prd.md](file:///Users/vlad/Developer/cloudlysis/aggregate/prd.md#L82-L151)). Tenant placement primitives and KV helper exist ([swarm.rs](file:///Users/vlad/Developer/cloudlysis/aggregate/src/swarm.rs#L5-L227)).
- **Gateway**: provides the platform ingress, authn/authz, and tenant-aware routing; it explicitly expects NATS KV-based tenant placement and hot reload in production ([gateway/prd.md](file:///Users/vlad/Developer/cloudlysis/gateway/prd.md#L13-L175)).
- **Projection**: consumes events, stores read models, and expects tenant-scoped query isolation and operational monitoring (consumer lag, checkpoints) ([projection/prd.md](file:///Users/vlad/Developer/cloudlysis/projection/prd.md#L7-L96)).
- **Runner**: executes sagas + effects, includes tenant assignment watching via NATS KV and tenant draining semantics ([tenant_placement.rs](file:///Users/vlad/Developer/cloudlysis/runner/src/tenant_placement.rs#L11-L104)) and exposes admin endpoints for drain/reload in its PRD ([runner/prd.md](file:///Users/vlad/Developer/cloudlysis/runner/prd.md#L199-L210)).

The control plane also adopts the proven **Admin UI UX + component library** from UltraBase’s control-plane admin UI, adapting screens and information architecture to Cloudlysis needs:
- Reusable UI components live under [ui/control-plane-admin/src/components/ui](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/components/ui).
- Example pages include [TenantsPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TenantsPage.tsx), [AdminUsersPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/AdminUsersPage.tsx), [AdminSessionsPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/AdminSessionsPage.tsx), [FleetPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/FleetPage.tsx), [TopologyPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TopologyPage.tsx), and [ObservabilityPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/ObservabilityPage.tsx).

---

## **Problem Statement**

Operating the platform without a unified control plane forces operators to:
- Use ad-hoc scripts, direct cluster access, or service-local admin endpoints
- Manage tenants, placements, and deployments without a consistent audit trail
- Correlate production incidents across services with incomplete dashboards and unsafe levels of access

The platform needs a control plane that:
- Centralizes **admin workflows** and **production operability**
- Enforces **least-privilege RBAC**, **step-up**, and **auditing**
- Provides a consistent, safe abstraction over **tenant placement**, **scale**, and **production operations**

---

## **Goals**

- Deliver an Admin UI with full admin management over:
  - users, sessions, roles/permissions
  - configuration (global + per-tenant)
  - definitions (aggregates, projections, sagas, effects, manifests)
  - scaling and production management (tenant placement, drains, migrations, deployments)
- Package production-grade monitoring:
  - metrics via VictoriaMetrics
  - logs via Loki
  - dashboards and alerting via Grafana (+ vmalert where used)
- Make production operations observable, auditable, and safe by default:
  - strong change logging + approvals where needed
  - idempotent operations + dry runs + rollback paths

---

## **Non-Goals**

- Re-implement node business logic (Aggregate / Projection / Runner) or platform ingress (Gateway).
- Replace NATS JetStream, libmdbx storage responsibilities, or per-service runtime concerns.
- Provide an arbitrary “general API gateway” for third-party upstreams.

---

## **Primary Users**

- **Platform Owner / SRE**: fleet operations, incident response, production change management.
- **Platform Admin**: tenant provisioning, RBAC, config/definition promotion.
- **Security Admin**: access reviews, session revocation, audit trails.
- **Support / On-call**: triage dashboards, logs/metrics correlation, safe mitigations (drain, disable, rollback).

---

## **Key Concepts**

### Control Plane Scope

- The control plane is the authoritative interface for production operations and admin management.
- The control plane uses node APIs, the Gateway, and NATS KV as its operational substrate rather than bypassing them.

### Tenant-Aware Operations

- All tenant-scoped operations are keyed by `tenant_id` (consistent with `x-tenant-id` usage across nodes and Gateway).
- Tenant placement is treated as a first-class “control plane state” (NATS KV-backed in production; file/static in development), consistent with existing code patterns ([swarm.rs](file:///Users/vlad/Developer/cloudlysis/aggregate/src/swarm.rs#L188-L226), [tenant_placement.rs](file:///Users/vlad/Developer/cloudlysis/runner/src/tenant_placement.rs#L41-L104)).

### Safe Change Management

- Mutating actions require explicit intent, are recorded in audit logs, and should be reversible where possible.
- All high-impact operations support:
  - validation and preflight checks
  - dry-run planning
  - idempotency keys
  - explicit rollback guidance

### Control Plane Components (In This Repo)

- **Admin UI (React)**:
  - Reuse UltraBase’s control-plane admin UI component system and interaction patterns, adapting routes and pages to Cloudlysis requirements ([components/ui](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/components/ui)).
  - The UI should prefer “table + detail pages + action dropdown + modals” patterns to keep ops workflows fast and consistent.
- **Control Plane API (BFF / Admin API)**:
  - A thin API layer that enforces RBAC, writes audit logs, and orchestrates multi-step operations (drain/migrate/rollout) as idempotent jobs.
  - Integrates with the Gateway for platform authn/authz and with node admin endpoints for operational actions.
- **Observability Stack**:
  - Version-controlled provisioning for Grafana dashboards/datasources, scrape configs for vmagent, and alert rules (vmalert or Grafana Alerting), modeled after UltraBase’s baseline ([observability/README.md](file:///Users/vlad/Developer/madapes/ultrabase/observability/README.md#L1-L47)).

---

## **Functional Requirements**

### 1) Admin IAM (Users, Sessions, Roles)

#### 1.1 Users

- CRUD users with lifecycle states:
  - invited (pending acceptance), active, suspended, disabled, deleted (tombstoned)
- Identity attributes:
  - email (primary), optional secondary identities
  - display name, avatar, metadata tags
  - auth methods enabled (password, OIDC providers), MFA state
- Administrative actions:
  - invite/resend invite
  - reset password flow initiation
  - force MFA reset / revoke recovery codes
  - disable login / suspend user
  - impersonation (break-glass, audited, time-boxed)
- Security constraints:
  - privileged actions require step-up / recent auth
  - sensitive events must be audit logged (who, what, when, why, from where)

#### 1.2 Sessions

- View active sessions and refresh token families:
  - by user, by tenant, by IP / geo, by device, by time range
- Revoke capabilities:
  - revoke a single session
  - revoke all sessions for a user
  - revoke all sessions for a tenant (incident response)
- Detection surfaces:
  - unusual session fanout (many sessions per user)
  - repeated failed logins / MFA failures
  - suspicious IP changes

#### 1.3 Roles & Permissions (RBAC)

- Roles are sets of permissions; assignments bind principals to roles in a scope.
- Scopes:
  - global (platform-level)
  - tenant-scoped
  - environment-scoped (dev/staging/prod) when applicable
- Required permission domains (minimum):
  - iam.users.* (create/update/suspend/delete)
  - iam.sessions.* (list/revoke)
  - iam.roles.* (create/update/assign)
  - tenants.* (create/update/archive)
  - configs.* (read/write/approve/apply)
  - definitions.* (read/write/validate/promote/rollback)
  - scale.* (view/apply/migrate/drain)
  - ops.* (deploy/rollback/restart/drain)
  - observability.* (view dashboards, manage alert rules)
  - audit.* (view/export)
- Role templates:
  - owner, admin, operator, support, read-only, security-admin, break-glass

---

### 2) Tenant Management

- Create, list, and archive tenants.
- Tenant status model:
  - provisioning, active, draining, migrating, degraded, suspended, archived
- Tenant metadata:
  - plan/tier, quotas, feature flags, contact + billing metadata, environment(s)
- Tenant operational actions:
  - trigger provisioning workflows (create streams/buckets, seed configs, create placement)
  - rotate tenant secrets (as definitions/config allow)
  - pause/resume workload (soft kill switch via config flags)

Tenant pages should mirror UltraBase’s “Tenant Overview + subpages” navigation patterns (example: [TenantsPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TenantsPage.tsx) and [TenantOverviewPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TenantOverviewPage.tsx)).

---

### 3) Configuration Management (Global + Per-Tenant)

#### 3.1 Config Model

- Config items are versioned, typed documents with:
  - scope (global / tenant / environment)
  - schema version
  - provenance (who/what wrote it)
  - effective date and rollout strategy
- Config must support:
  - validation against a schema
  - diff view (previous vs next)
  - staged rollout (preview → apply)
  - rollback to a prior version

#### 3.2 Node-Related Configuration

Required config surfaces (minimum):
- **Gateway**: routing/placement sources, auth policies, rate limits (see routing expectations in [gateway/prd.md](file:///Users/vlad/Developer/cloudlysis/gateway/prd.md#L154-L175)).
- **Aggregate / Projection / Runner**:
  - shard identifiers and tenant allowlists/placement settings
  - drain/reload toggles and safety thresholds
  - resource limits / concurrency caps

---

### 4) Definition Management (System “Blueprints”)

Definitions are the declarative “what the platform is” and “what runs” layer: aggregates, projections, sagas, effect providers, and any manifests that tie runtime-function programs to entity types.

Required capabilities:
- Upload/edit versioned definitions with:
  - validation (schema + semantic checks)
  - “impact analysis” (which tenants/services are affected)
  - promotion workflow (dev → staging → prod)
- Change controls:
  - approvals (role-based) for production promotion
  - emergency rollback path (one-click revert to last-known-good definition bundle)
- Tenant overrides:
  - allow per-tenant definition overrides only when explicitly permitted by policy

The control plane must present definitions in a way that maps to the node runtime responsibilities:
- Aggregates and deterministic decide/apply programs ([aggregate/prd.md](file:///Users/vlad/Developer/cloudlysis/aggregate/prd.md#L155-L160))
- Projections and deterministic project programs ([projection/prd.md](file:///Users/vlad/Developer/cloudlysis/projection/prd.md#L36-L55))
- Runner sagas and effect provider manifests ([runner/prd.md](file:///Users/vlad/Developer/cloudlysis/runner/prd.md#L41-L57))

---

### 5) Scale Management (Tenant Placement, Shards, Fleet)

#### 5.1 Placement Model

- Placement is modeled as:
  - a set of nodes/shards and their attributes (labels, capacity, region)
  - tenant → shard assignments per service kind (Aggregate, Projection, Runner, optionally Gateway when relevant)
- Control plane supports both:
  - static placement (development)
  - dynamic placement (production) backed by NATS KV (consistent with existing client patterns in [swarm.rs](file:///Users/vlad/Developer/cloudlysis/aggregate/src/swarm.rs#L79-L227))

#### 5.2 Tenant Migration

- Provide guided migration planning and execution:
  - show current assignment, target assignment, and a sequenced action plan
  - execute “graceful drain → update placement → reload” style plans (see [plan_graceful_tenant_migration](file:///Users/vlad/Developer/cloudlysis/aggregate/src/swarm.rs#L41-L65))
- Migration safety:
  - require explicit confirmation and reason
  - block if draining is unsafe (inflight work too high, storage unhealthy, consumer lag too high)
  - time-box and alert if drains do not converge

#### 5.3 Fleet View

- Fleet inventory:
  - nodes (labels, region, capacity, version)
  - services (replicas, image version, health)
  - per-node and per-service load indicators (CPU/mem, request rate, consumer lag)
- Operator actions:
  - scale replicas, restart services, cordon/drain nodes (when supported by orchestrator)

UX should align with the UltraBase “Fleet” and “Topology” navigation patterns ([FleetPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/FleetPage.tsx), [TopologyPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TopologyPage.tsx)).

---

### 6) Production Operations (Deployments, Maintenance, Safety)

#### 6.1 Deployments

- Manage deployable artifacts per service (Aggregate/Gateway/Projection/Runner) with:
  - environment-specific rollout policies
  - canary/rolling deploy support (when orchestrator supports it)
  - automatic health checks gates and rollback triggers
- Track releases:
  - “what is running where” (service version matrix)
  - change log links and approvals

#### 6.2 Maintenance Operations

- Drain operations:
  - tenant drain (stop acquiring new work, finish inflight; required by Runner semantics in [TenantGate](file:///Users/vlad/Developer/cloudlysis/runner/src/tenant_placement.rs#L106-L200))
  - node drain (aggregate tenant ranges, projection consumers, runner workers)
- Replay / rebuild operations:
  - projection rebuild triggers (dangerous, must be guarded and audited)
  - workflow replay controls (reset checkpoints only with explicit intent)

#### 6.3 Incident Response Toolkit

- “Safe switches”:
  - per-tenant kill switch (disable commands/effects via config)
  - global degrade modes (rate limit reductions, disable expensive features)
- Run actions:
  - revoke sessions at scale
  - freeze deployments
  - trigger drain/migrate with guided plan

---

### 7) Observability (VictoriaMetrics + Loki + Grafana) and Dashboards

#### 7.1 Stack Requirements

Adopt a production-ready stack consistent with UltraBase’s operational baseline:
- **VictoriaMetrics** for metrics storage and Prometheus-compatible query
- **vmagent** for scraping and remote_write
- **Grafana** for dashboards and alert routing
- **Loki** (+ optional **Promtail**) for logs
- Optional **vmalert** for rule evaluation against VictoriaMetrics

UltraBase’s observability design is a direct reference implementation to mirror and adapt:
- Stack overview and conventions: [observability/README.md](file:///Users/vlad/Developer/madapes/ultrabase/observability/README.md#L1-L47)
- Provisioned dashboards and datasources: [grafana provisioning](file:///Users/vlad/Developer/madapes/ultrabase/observability/grafana/provisioning)

#### 7.2 Metrics Conventions

- Every service exports `/metrics` in Prometheus format.
- Required labels:
  - `service` (stable, low cardinality)
  - `env` (dev/staging/prod)
  - `tenant_id` only where safe and bounded; avoid tenant_id on high-frequency per-request series unless cardinality is controlled.
- HTTP metrics must avoid unbounded `path` cardinality; prefer route templates (pattern-based paths).

Tenant-aware metrics guidelines:
- Prefer **tenant-only aggregates** for “who is hurting us?” views:
  - `..._requests_total{tenant_id,service,status_class}` (no `path`)
  - `..._request_duration_seconds{tenant_id,service}` (no `path`, limited bucket count)
- Prefer **route-only aggregates** for “what endpoint is hurting us?” views:
  - `..._requests_total{service,path,status}` (no `tenant_id`)
- Where per-tenant and per-route both matter, implement a **top-k sampling** policy:
  - emit `(tenant_id,path)` series only for top N tenants, or only for a fixed allowlist of routes.

#### 7.3 Required Dashboards (Production)

Minimum set of dashboards (provisioned on startup):
- **Platform — Operations overview**
  - `up` for core services and observability stack
  - RPS, 4xx/5xx ratio, p95/p99 latency per service
  - saturation indicators (CPU/mem, inflight, queue depth)
- **Platform — HTTP detail**
  - per-service request breakdown by route template, method, status
  - top failing paths and latency outliers
- **Platform — Logs**
  - Loki stream filtering by `service`, `tenant_id` (where present), and correlation identifiers
- **Platform — Event bus / JetStream**
  - consumer lag, redeliveries, ack latency, stream storage pressure
- **Platform — Workers (Runner)**
  - outbox depth, effect latency, poison message counts, schedules backlog
- **Platform — Storage (libmdbx)**
  - DB size growth, write stalls, fsync latency (where exported), disk usage
- **Platform — Cluster / Orchestrator**
  - node health, container restarts, placement distribution by tenant range

Dashboards should be modeled after UltraBase’s default set (for structure, not content), e.g. [ultrabase-operations.json](file:///Users/vlad/Developer/madapes/ultrabase/observability/grafana/provisioning/dashboards/default/ultrabase-operations.json) and [ultrabase-http-detail.json](file:///Users/vlad/Developer/madapes/ultrabase/observability/grafana/provisioning/dashboards/default/ultrabase-http-detail.json).

Additional production-operability dashboards (chosen and adapted):
- **Platform — Noisy Neighbor & Tenant Health**
  - Purpose: identify a tenant causing cluster instability (attack, runaway job, bad config) and quickly pivot all panels to that tenant.
  - Panels (minimum):
    - Top tenants by Gateway RPS (topk of tenant-only request counters).
    - Tenant latency distribution (p95/p99 per tenant) from tenant-only latency histograms.
    - Tenant error ratio (5xx and 429) per tenant.
    - Aggregate in-flight commands by tenant (already exported: `aggregate_in_flight_commands{tenant_id}`).
    - Projection processing error rate by tenant (from `projection_processing_errors_total{tenant_id,view_type}` aggregated per tenant).
    - Loki logs panel with a `tenant_id` variable selector; selecting a tenant syncs RPS/latency/errors + logs.
  - Required instrumentation:
    - Gateway must expose **tenant-level** HTTP counters/histograms (tenant + status class + service, without `path`) in addition to existing route-level metrics.

- **Platform — API Regression & Deployment**
  - Purpose: determine whether a newly rolled out image caused regressions, and correlate changes with deployment events.
  - Panels (minimum):
    - Error rate comparison “old vs new” by `service` and `version` (or `image_tag`) labels.
    - Latency comparison “old vs new” (p95/p99) per service.
    - Restart / flapping rate per service (container restarts, crash loops).
    - Dependency latency correlation:
      - Gateway request duration vs Aggregate command duration vs Projection processing duration vs Runner effect latency.
    - Loki “new errors” panel:
      - errors seen in the last 10m that were not present in the prior 60m window, grouped by `service`.
    - Deployment annotations:
      - vertical markers when Swarm service updates started/finished (via annotations or a deploy event metric).
  - Required instrumentation:
    - Every service exports a `*_build_info{service,version,git_sha}` gauge (value=1) or equivalent, and scrape relabeling adds `image_tag` where possible.
    - Control plane emits deployment annotations/events (or pulls them from the orchestrator and writes to Grafana annotations).

- **Platform — Storage & Event Bus Bottlenecks**
  - Purpose: debug timeouts when the API is “up” but underlying storage/eventing is saturated (the Cloudlysis equivalent of DB firefighting).
  - Panels (minimum):
    - NATS/JetStream health:
      - stream storage pressure, publish/ack latency, consumer lag, redeliveries.
    - Projection lag and throughput:
      - events processed rate, processing duration, error rate.
    - Aggregate write-path pressure:
      - command duration, version conflicts, in-flight commands, tenant errors.
    - Runner pressure:
      - outbox dispatch failure rate, effect timeout rate, deadletter writes.
    - Disk saturation on nodes hosting libmdbx:
      - disk usage, read/write latency, IOPS; correlate with spikes in command/query latency.
    - Optional Postgres/Autobase panels only when a managed DB backs any control-plane metadata:
      - pool saturation, replica lag, slow queries, long transactions.
  - Required instrumentation:
    - Ensure JetStream metrics are scraped (NATS server `/varz` exporter or native Prometheus endpoint depending on deployment).
    - Ensure node-level disk/IO metrics are scraped (node exporter / cadvisor / equivalent).

- **Platform — Infrastructure Exhaustion**
  - Purpose: detect node/resource pressure earlier than raw CPU% and catch observability blind spots.
  - Panels (minimum):
    - CPU/memory pressure (PSI) per node (when available), plus load average and CPU saturation.
    - OOM kill tracker across the cluster.
    - Disk usage + IO wait/latency on data volumes (libmdbx, Loki, VictoriaMetrics).
    - vmagent health:
      - scrape error rate, remote_write errors, queue backlog.
    - Loki ingestion health:
      - dropped log lines (promtail) and ingestion errors (loki).
    - Swarm task hygiene:
      - desired_state vs current_state mismatches, orphaned tasks, restart loops.
  - Required instrumentation:
    - node exporter / cadvisor (or equivalent) must be part of the production scrape plan.
    - promtail (or alternative) must expose drop/error metrics when logs are enabled.

#### 7.4 Alerting Requirements

Minimum alert classes:
- Availability:
  - service down (`up == 0`)
  - scrape failures, vmagent remote_write errors
- Reliability:
  - sustained elevated 5xx ratio
  - sustained elevated p95 latency per service
- Backlogs:
  - JetStream consumer lag above threshold
  - Runner outbox depth above threshold
- Data safety:
  - disk usage near full (nodes hosting libmdbx)
  - abnormal restart loops
- Security:
  - login anomaly detection signals (where instrumented)
  - suspicious spike in session revocations / failed MFA

Alert rules can follow UltraBase’s approach of version-controlled rules in YAML (reference: [alerts/](file:///Users/vlad/Developer/madapes/ultrabase/observability/alerts)).

#### 7.5 Control Plane → Observability Linking

The Admin UI must embed or deep-link into observability tools:
- per-tenant and per-service quick links to Grafana dashboards and Loki queries
- incident triage shortcuts (operations overview → HTTP detail → logs)

This mirrors UltraBase’s “observability links JSON” concept ([observability/README.md](file:///Users/vlad/Developer/madapes/ultrabase/observability/README.md#L65-L75)), but adapted to Cloudlysis services and dashboards.

---

### 8) Audit, Compliance, and Change History

- Audit log is an append-only stream of security and operations events:
  - authentication and session events
  - RBAC changes and permission grants
  - config/definition changes and promotions
  - scaling, drain, and migration operations
  - deployments and rollbacks
- Audit log must support:
  - search and export (bounded and access controlled)
  - correlation to production incidents (request ids, trace ids)
  - retention policy controls

---

### 9) Control Plane API Surface (Admin API)

The control plane requires a stable API surface for the Admin UI and automation.

Minimum API capabilities:
- **Idempotent jobs for multi-step operations**:
  - every mutating operation returns a `job_id`, supports polling and cancellation, and records a full execution trace in the audit log.
- **Preflight endpoints**:
  - validate an intended change and return a plan (and “would-change” diff) without applying it.
- **RBAC-first access model**:
  - all endpoints enforce permission checks at the API boundary (UI is not trusted).

Minimum endpoint groups:
- `/admin/v1/iam/*` (users, roles, assignments, sessions)
- `/admin/v1/tenants/*` (tenants lifecycle, status, metadata)
- `/admin/v1/config/*` (versioned config, diff, apply, rollback)
- `/admin/v1/definitions/*` (bundles, validate, promote, rollback)
- `/admin/v1/scale/*` (placement, migrations, drain status)
- `/admin/v1/ops/*` (deployments, rollbacks, service actions)
- `/admin/v1/observability/*` (links, saved queries, dashboard registry)
- `/admin/v1/audit/*` (search, export)

Authentication/authorization integration:
- Prefer using the **Gateway** as the system of record for admin identities and sessions, with the control plane API validating requests using Gateway-issued tokens and enforcing platform-specific permissions.

---

### 10) Secrets and Credentials Management

The control plane must treat secrets as first-class operational data with strict handling.

Requirements:
- Secret values must never be logged and must be redacted in UI/API responses.
- Secrets must support:
  - creation and rotation workflows
  - scoped access (global/tenant/environment)
  - staged rollout (write new → verify → promote → retire old)
- Rendering rules:
  - after creation, secret plaintext must not be retrievable unless explicitly enabled by policy (default: write-only).
- Integrations:
  - support referencing secrets from config/definitions without embedding values (secret refs).

---

### 11) Backups, Restore, and Disaster Recovery (Production Operability)

The control plane must provide explicit visibility and guardrails for data safety operations.

Minimum requirements:
- **Backup status**:
  - show last successful backup timestamps per critical store (metadata DB, NATS state if applicable, Grafana provisioning state as code, tenant placement/config stores).
- **Restore readiness**:
  - preflight checks that validate a restore plan (target environment, versions, dependencies).
- **Operational playbooks**:
  - link to the exact restore procedure and post-restore verification checklist.
- **Key rotation**:
  - explicit workflows and audit logs for rotating signing keys, service credentials, and secret backends.

This should align with the platform’s existing operational patterns (e.g., the explicit “restore / post-restore checks” concept used in UltraBase observability docs).

---

## **Admin UI Requirements (Information Architecture + UX)**

### Navigation (Minimum)

Left navigation sections:
- Overview
- Tenants
- Users
- Sessions
- Roles & Permissions
- Config
- Definitions
- Scale & Placement
- Deployments
- Observability
- Audit Log
- Settings

### Page Patterns (Reuse UltraBase UI)

Adopt the UltraBase component system and page layout patterns:
- Layout, styling tokens, UI primitives: [components/ui](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/components/ui)
- Table + search + action dropdown pattern: [TenantsPage](file:///Users/vlad/Developer/madapes/ultrabase/ui/control-plane-admin/src/pages/TenantsPage.tsx#L94-L203)

Required page types:
- List pages:
  - searchable table, bulk actions, row actions menu, status pills, empty states
- Detail pages:
  - header with primary actions (drain, migrate, rollback)
  - sub-nav tabs for domain-specific views
- Mutation flows:
  - modal confirmation + explicit reason entry for high-impact changes
  - toast notifications and “busy” state handling consistent with UltraBase patterns

### Tenant Detail Subpages (Minimum)

- Overview (status, assignments, SLO highlights)
- Placement (per service: Aggregate/Projection/Runner)
- Health (node readiness and dependency checks)
- Config (effective config + diffs)
- Definitions (applied definition bundle + version)
- Activity (audit trail filtered to tenant)
- Observability (embedded links / panels)

---

## **Non-Functional Requirements**

- **Security**:
  - strict RBAC everywhere; deny-by-default
  - audit every privileged operation
  - step-up for sensitive actions
  - CSRF protection for browser sessions
  - safe secret handling (no secret values rendered after creation unless explicitly permitted)
  - allowlist outbound integrations (Grafana/Loki/VM URLs, orchestration API endpoints) to prevent SSRF-style abuse
- **Reliability**:
  - control plane operations are idempotent and resilient to partial failures
  - operations have clear “current state” and do not rely on UI assumptions
- **Performance**:
  - list pages paginate and filter server-side for large fleets
  - dashboards load with bounded query costs and controlled label cardinality
- **Operability**:
  - control plane itself must be observable (metrics/logs, dashboards, alerts)
  - every operation must surface preflight checks and post-conditions

---

## **Open Questions / Design Constraints (To Resolve During Implementation)**

- Where does the source of truth live for:
  - users/sessions/roles (Gateway vs control-plane backing store)?
  - configs/definitions (NATS KV vs database vs GitOps)?
- How should production promotions be modeled:
  - environment branches, approval workflow, and rollback semantics?
- What orchestrator is the production baseline (Docker Swarm per existing PRDs, or will Kubernetes be introduced)?
 - Where should the job/execution state for long-running operations live:
   - embedded in the control plane API process, durable store, or NATS workflows?