cloudlysis/GATEWAY_TRANSPORT_PLAN.md

# Gateway Transport Plan

## Purpose
Standardize and optimize how the Gateway communicates with Aggregate, Projection, and Runner, and how nodes communicate via NATS JetStream, under these principles:
- Simplicity (few patterns, minimal bespoke conventions)
- Ease of operation (consistent health/ready/metrics, consistent failure modes)
- Frugality (bounded connections, bounded fanout, low overhead)
- High performance (low tail latency, backpressure-aware, predictable routing)
- Safety (tenant isolation, deny-by-default authz, consistent context propagation)

## Non-Negotiable Rules (Global)
- Every cross-service request MUST carry tenant + trace context.
- Every transport path MUST have explicit timeouts/deadlines and bounded retries.
- Every milestone below is “stop-the-line” gated:
  - All tasks completed
  - All tests passing
  - Workspace lint/format/type checks passing
  - Required integration tests for the milestone passing (when gated by env, they must be runnable and documented)

## Current State (Baseline)
- Gateway → Aggregate: gRPC command submission
- Gateway → Projection: HTTP query proxy (`/v1/query/*`)
- Gateway → Runner: HTTP proxy for admin endpoints (`/admin/runner/*`)
- Nodes ↔ NATS JetStream: events/workflow streams with headers for tenant/correlation/trace (now more consistent)

## Target Architecture (End State)
- Edge contract (clients ↔ Gateway): HTTP/JSON (stable, debuggable, browser + ops friendly)
- Internal RPC (Gateway ↔ services): gRPC for Aggregate + Projection + Runner (single internal RPC stack)
- Async/event backbone: NATS JetStream remains for event/work distribution
- `shared` is the single source of truth for:
  - Header names and propagation rules
  - Trace parsing/validation rules (`traceparent`, `trace-id`)
  - Request context representation (tenant/correlation/trace)

## Definitions
### Request Context
Fields that must be consistently propagated:
- `tenant_id` (HTTP: `x-tenant-id`, NATS: `tenant-id`)
- `correlation_id` (HTTP: `x-correlation-id`, NATS: `x-correlation-id` and `correlation-id`)
- `traceparent` (HTTP: `traceparent`, NATS: `traceparent`)
- `trace_id` (derived from `traceparent` or provided explicitly; NATS: `trace-id`)
- `request_id` (HTTP: `x-request-id`, optional for NATS)

### Standard Health Endpoints (per service)
- `GET /health` liveness
- `GET /ready` readiness (includes tenant gating if applicable)
- `GET /metrics` Prometheus

## Milestone 0: Transport Contract Lock-in (Context + Headers Everywhere)

### Goal
Make context propagation and header naming consistent and enforceable across HTTP, gRPC, and NATS, including “background” Gateway calls (health checks, rebalance probes).

### Exit Criteria
- A single shared contract exists for header names and trace parsing.
- Gateway injects context into all upstream calls (including rebalance/health probes).
- Aggregate/Projection/Runner consistently emit/consume the standard context on all transport paths they own.
- Unit tests prove propagation behavior for each transport.
- `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace` all pass.

### Tasks
- [ ] Standardize header constants in `shared` and remove string literals from Gateway and nodes where feasible.
- [ ] Add `shared` helpers for:
  - HTTP extract/inject
  - gRPC metadata extract/inject
  - NATS header extract/inject
- [ ] Gateway: ensure context is injected into:
  - gRPC upstream requests to Aggregate
  - HTTP upstream requests to Projection
  - Runner admin proxy requests
  - Any “probe” calls (rebalance gates, fleet snapshots, health checks)
- [ ] Projection/Runner/Aggregate: ensure NATS published messages include:
  - `tenant-id`
  - `x-correlation-id` + `correlation-id`
  - `traceparent`
  - `trace-id` (derived when possible)
- [ ] Add transport-level tests:
  - [ ] Gateway gRPC path: incoming context → upstream metadata → response metadata preserved
  - [ ] Gateway HTTP proxy path: incoming context → upstream headers preserved
  - [ ] NATS publish path: produced headers contain expected keys/values

### Required Tests
- Unit tests for shared parsing/derivation utilities
- Existing per-crate test suites
- At least one per-service “transport contract” test verifying headers are present and correct

## Milestone 1: Internal RPC Standardization (Projection via gRPC)

### Goal
Eliminate Gateway → Projection HTTP proxy as the default path by introducing an internal gRPC Query service, keeping HTTP optional for human/debug use.

### Exit Criteria
- A Projection gRPC service exists for query execution.
- Gateway routes queries to Projection via gRPC by default.
- Authorization semantics remain enforced in Gateway (deny-by-default).
- Response shapes are stable and match the existing UI expectations.
- All tests pass, including new gRPC query integration tests.

### Tasks
- [ ] Define protobuf API: `projection.gateway.v1.QueryService`
  - [ ] Request includes tenant + view + query payload and metadata
  - [ ] Response includes result payload and standard context propagation
- [ ] Implement Projection gRPC server:
  - [ ] Parse tenant/view/query
  - [ ] Execute query against current projection storage/query engine
  - [ ] Enforce tenant scope
- [ ] Implement Gateway gRPC client path for queries:
  - [ ] Routing by tenant to Projection endpoint
  - [ ] Deadlines, bounded retries (idempotent only)
  - [ ] Context propagation (tenant/correlation/trace)
- [ ] Keep HTTP `/v1/query/*`:
  - [ ] Either route to internal gRPC implementation or keep as legacy/debug endpoint
- [ ] Add tests:
  - [ ] Gateway query authz + forwarding via gRPC
  - [ ] Projection gRPC query contract tests for tenant isolation

### Required Tests
- New gRPC QueryService tests (unit + integration)
- Existing query/authz tests in Gateway
- Workspace fmt/clippy/test

## Milestone 2: Internal RPC Standardization (Runner Admin via gRPC)

### Goal
Replace `/admin/runner/*` HTTP proxying with a first-class gRPC admin service for Runner operations.

### Exit Criteria
- Runner exposes a gRPC admin service for the admin surface required by Control/Gateway.
- Gateway uses gRPC to call Runner admin APIs.
- Authentication/authorization remains in Gateway; Runner trusts Gateway boundary.
- Admin operations are idempotent where appropriate and include audit hooks where required.
- All tests pass and include negative/tenant-spoof cases.

### Tasks
- [ ] Define protobuf API: `runner.admin.v1.RunnerAdmin`
  - [ ] Drain/resume/status/reload/tenant-scoped controls
  - [ ] Standard error mapping
- [ ] Implement Runner gRPC admin server:
  - [ ] Tenant gating enforced for tenant-scoped operations
  - [ ] Readiness/drain semantics aligned with platform contracts
- [ ] Implement Gateway gRPC client integration:
  - [ ] Route to Runner endpoint via routing table
  - [ ] Enforce authz rights (e.g. `runner.admin`)
  - [ ] Context propagation
- [ ] Keep HTTP `/admin/*` in Runner optional:
  - [ ] Either remove Gateway proxy usage or keep for direct debugging behind secure network
- [ ] Tests:
  - [ ] Gateway: admin calls rejected without rights
  - [ ] Gateway: tenant spoof attempts rejected
  - [ ] Runner: idempotency and drain semantics validated

### Required Tests
- gRPC RunnerAdmin unit/integration tests
- Gateway proxy-to-gRPC tests
- Workspace fmt/clippy/test

## Milestone 3: Connection + Retry Policy Unification (Performance + Frugality)

### Goal
Make upstream connection management and retry behavior consistent and bounded across Gateway and nodes.

### Exit Criteria
- Gateway maintains bounded upstream connection pools for gRPC endpoints.
- All gRPC calls have deadlines; retries are only for idempotent operations.
- All probe/fanout calls are bounded and do not cause thundering herds.
- Load/soak tests show stable behavior under partial failure.

### Tasks
- [ ] Implement a Gateway upstream channel pool:
  - [ ] LRU bounded by max endpoints
  - [ ] TTL/eviction strategy
  - [ ] Fast path reuse under load
- [ ] Standardize retry profiles:
  - [ ] Read-only: short retry with jitter
  - [ ] Mutations: no automatic retry unless idempotency key present
- [ ] Standardize timeouts:
  - [ ] Edge timeout limits
  - [ ] Internal per-service deadlines
- [ ] Fanout controls:
  - [ ] Concurrency limiters for fleet snapshot/probes
  - [ ] Cache results where safe (short TTL)

### Required Tests
- Unit tests for pool eviction/TTL
- Gateway integration tests for deadline propagation
- Gated load tests (document env + how to run)

## Milestone 4: Transport Simplification Cleanup (Remove Legacy Paths)

### Goal
Remove or de-prioritize legacy HTTP internal paths so the “happy path” uses: HTTP edge → Gateway → gRPC internal → NATS async.

### Exit Criteria
- Gateway no longer depends on HTTP for Projection queries or Runner admin.
- Legacy endpoints are either removed or explicitly marked “debug-only” and not used by Gateway/Control.
- All operational playbooks rely on standardized endpoints.

### Tasks
- [ ] Remove Gateway’s HTTP query proxy usage (or keep only as compatibility shim).
- [ ] Remove Gateway’s runner admin HTTP proxy usage (or keep only as compatibility shim).
- [ ] Ensure Control UI + Control API use the standardized Gateway surfaces.
- [ ] Harden metrics and health probes to always carry context.

### Required Tests
- End-to-end smoke tests (gated)
- Workspace fmt/clippy/test

## Verification Commands (Required at Each Milestone)
- `cargo fmt --check`
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test --workspace`
- `npm ci && npm run lint && npm run typecheck && npm run test && npm run build` (in `control/ui`)

## Notes / Constraints
- Do not break wire compatibility for NATS subjects or event payloads; evolve via optional fields and tolerant decoding.
- Keep tenant isolation rules enforced at the Gateway boundary and re-validated at nodes where it is safety-critical.