cloudlysis/NATS_TRANSPORT_PLAN.md

# NATS Transport Plan

## Purpose
Standardize and optimize how nodes (Aggregate, Projection, Runner, Gateway where applicable) use NATS JetStream and NATS KV, under these principles:
- Simplicity (few primitives, consistent naming, minimal per-service divergence)
- Ease of operation (predictable streams/consumers, clear runbooks, easy debugging)
- Frugality (bounded consumers, bounded in-flight work, minimal churn, minimal storage)
- Low resource usage (stable durable consumers, controlled ack waits, limited fanout)
- High performance (high throughput, low tail latency, reliable backpressure)
- Safety (tenant isolation, idempotency, deterministic replay, poison handling)

## Non-Negotiable Rules (Global)
- Every JetStream stream/consumer MUST have an explicit contract:
  - name, subjects, retention, storage, replication, max sizes
  - ack policy, ack wait, max deliver, max in flight
- Every node MUST run with bounded work:
  - bounded pull batch sizes
  - bounded concurrency
  - bounded retry/backoff
- Every message MUST be tenant-scoped in subject and/or headers.
- Every milestone below is “stop-the-line” gated:
  - all tasks completed
  - all tests passing
  - workspace lint/format checks passing
  - required NATS-gated integration tests for the milestone passing (when gated by env)

## Current State (Baseline)
- Streams:
  - `AGGREGATE_EVENTS` (Aggregate publishes, Projection/Runner consume)
  - `WORKFLOW_COMMANDS`, `WORKFLOW_EVENTS` (Runner)
- Subject conventions:
  - Aggregate events: `tenant.<tenant_id>.aggregate.<aggregate_type>.<aggregate_id>`
  - Defaults often use filters like `tenant.*.aggregate.*.*`
- Durable consumers:
  - Projection uses a durable name (configurable)
  - Runner uses configurable durable prefix per role
  - Aggregate had ad-hoc fetch consumer risks; now mitigated with unique consumer names per fetch
- Headers:
  - Tenant + correlation + trace headers exist but were historically inconsistent; shared utilities now exist

## Target Architecture (End State)
- A single “NATS wire protocol” contract shared across services:
  - subject naming
  - required headers (tenant/correlation/trace)
  - message envelope compatibility rules (tolerant decoding, optional fields)
- Stable, minimal set of JetStream streams:
  - one stream per message class (aggregate events, workflow commands, workflow events)
  - no per-tenant streams unless there is a strong operational reason
- Stable, limited consumers:
  - durable consumers for long-lived processors (Projection, Runner)
  - ephemeral consumers only for bounded ad-hoc operations (Aggregate fetch), always unique + best-effort deletion
- Uniform backpressure + reliability defaults:
  - explicit ack
  - bounded `max_ack_pending` and application-level concurrency
  - bounded redelivery via `max_deliver` + poison policy

## Definitions
### Message Context (Headers)
Standard headers for NATS published messages:
- `tenant-id` (required)
- `x-correlation-id` and `correlation-id` (required for any request-derived message; generated if missing)
- `traceparent` (optional but recommended; generated/propagated if present upstream)
- `trace-id` (optional; derived from traceparent when possible)
- `Nats-Msg-Id` (required for idempotent publish/dedupe when applicable)

### Subject Naming Rules
- Tenant-first prefix: `tenant.<tenant_id>.…`
- Stable message class token:
  - `aggregate` for domain events
  - `effect`, `effect_result`, `workflow`, `workflow_event` for Runner
- No ambiguous wildcard publishing:
  - producers publish concrete subjects only
  - consumers may filter with wildcards

### Consumer Naming Rules
- Durable consumer names must be stable and collision-free:
  - include role + mode + optional view/saga name + shard/group
- Ephemeral consumer names must be unique per operation:
  - include tenant + purpose + uuid
  - must be deleted best-effort when operation completes

## Milestone 0: NATS Wire Contract Lock-in (Names, Headers, Envelopes)

### Goal
Make the NATS/JetStream wire contract explicit and enforced in code so all producers/consumers interoperate safely across scale-out and rolling restarts.

### Exit Criteria
- `shared` exposes NATS header constants and helpers for inject/extract/derive.
- All producers set required headers consistently.
- All consumers tolerate unknown fields and missing optional fields.
- A single, documented subject naming convention is enforced in code (builder functions).
- Workspace fmt/clippy/tests pass.

### Tasks
- [ ] Centralize NATS header constants and helpers in `shared`:
  - [ ] inject headers for publish (tenant, correlation, trace)
  - [ ] extract headers on receive (best-effort)
  - [ ] derive `trace-id` from `traceparent`
- [ ] Aggregate:
  - [ ] Ensure event publishing always sets `tenant-id`, correlation headers, trace headers
  - [ ] Ensure `Nats-Msg-Id` strategy is correct for idempotency/dedupe (document and test)
- [ ] Projection:
  - [ ] Ensure EventEnvelope decoding remains tolerant (unknown fields ignored, optional IDs supported)
  - [ ] Ensure correlation/trace context is carried into spans/metrics consistently
- [ ] Runner:
  - [ ] Ensure publish paths include correlation/trace headers consistently for commands and results
  - [ ] Ensure outbox metadata → NATS headers mapping is consistent and tested
- [ ] Tests:
  - [ ] Unit tests for header injection/extraction in `shared`
  - [ ] Per-service unit tests asserting produced headers include required keys

### Required Tests
- `cargo fmt --check`
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test --workspace`

## Milestone 1: Stream Configuration Standardization (Retention, Limits, Storage)

### Goal
Make stream configs consistent, explicit, and operationally sane across environments (dev → prod), minimizing surprise and preventing runaway resource usage.

### Exit Criteria
- Stream config for each stream is explicitly defined and validated at startup.
- Limits (max messages/bytes/age) are explicit and have defaults.
- Duplicate windows and dedupe behavior are explicit and tested.
- A “no destructive changes on startup” policy is enforced (create if missing; do not silently replace).

### Tasks
- [ ] Define a single “stream config policy” module per service (or shared helper):
  - [ ] `AGGREGATE_EVENTS` subjects + retention policy
  - [ ] `WORKFLOW_COMMANDS` subjects + retention policy
  - [ ] `WORKFLOW_EVENTS` subjects + retention policy
- [ ] Standardize defaults:
  - [ ] retention: limits appropriate for replay + rebuild
  - [ ] `duplicate_window` aligned with producer idempotency strategy
  - [ ] storage type and replication policy documented and configurable
- [ ] Add startup validations:
  - [ ] verify stream exists and matches required subject set (compatible superset allowed)
  - [ ] verify required ack/dedupe assumptions hold
- [ ] Add tests that parse and validate configs without NATS.

### Required Tests
- Unit tests for stream config builders
- Existing crate tests

## Milestone 2: Consumer Policy Standardization (Ack, Backpressure, Poison)

### Goal
Make consumption reliable and cheap under load by standardizing ack policy, concurrency, and poison/deadletter handling.

### Exit Criteria
- All long-lived consumers use explicit ack with consistent `ack_wait`, `max_deliver`, `max_ack_pending`.
- Application concurrency is bounded and tied to `max_in_flight`.
- Poison policy is consistent:
  - after `max_deliver`, term + deadletter/quarantine record is written
- Replay behavior is deterministic on restart (checkpoint-based where applicable).

### Tasks
- [ ] Define standard consumer config defaults:
  - [ ] `AckPolicy::Explicit`
  - [ ] `ack_wait` default + env override
  - [ ] `max_deliver` default + env override
  - [ ] `max_ack_pending` tied to application concurrency
- [ ] Projection:
  - [ ] Ensure durable consumer naming is collision-free in all modes (Single vs PerView)
  - [ ] Ensure checkpoint gates ack correctly (skip still acks)
  - [ ] Ensure poison policy writes durable records and terminates reliably
- [ ] Runner:
  - [ ] Ensure saga/effect consumers use consistent durable naming + deliver groups when scaling out
  - [ ] Ensure outbox relay preserves exactly-once semantics via dedupe keys + idempotent publish
- [ ] Aggregate:
  - [ ] Ensure ad-hoc fetch consumer is bounded (timeouts) and unique per operation (already required)
  - [ ] Ensure best-effort cleanup is performed and cannot delete unrelated consumers
- [ ] Tests:
  - [ ] Unit tests for consumer name generation (sanitization + uniqueness)
  - [ ] NATS-gated tests for ack/redelivery/poison behavior (must be runnable with env flag)

### Required Tests
- Workspace fmt/clippy/tests
- NATS-gated integration tests for:
  - redelivery idempotency
  - poison termination behavior
  - scale-out with deliver group (where supported)

## Milestone 3: Connection Management + Failure Semantics (Operational Frugality)

### Goal
Make NATS connection handling stable under partial failure while minimizing resource churn and cascading outages.

### Exit Criteria
- One NATS connection per process (or bounded pool only if justified).
- Reconnect/backoff policy is explicit and consistent.
- Circuit breaker behavior is consistent (when used), and health/ready reflect NATS state correctly.
- No busy-looping on NATS outages.

### Tasks
- [ ] Standardize connection options:
  - [ ] reconnect delays/backoff
  - [ ] max reconnect attempts or “infinite with backoff” strategy (explicit)
  - [ ] request timeouts around JetStream operations
- [ ] Standardize readiness semantics:
  - [ ] `ready=false` when NATS is unavailable and the node depends on it
  - [ ] `health` stays “process alive” but reports NATS connectivity in payload
- [ ] Add “fast fail” mode for tests and dev (avoid 30x retries when env not set).
- [ ] Tests:
  - [ ] unit tests for backoff behavior (where possible)
  - [ ] gated integration test: temporary NATS outage does not crash-loop and recovers

## Milestone 4: Multi-Tenant Scale-Out Guarantees (Collision-Free + Predictable)

### Goal
Guarantee safe multi-replica behavior: no consumer collisions, no duplicate side effects, predictable throughput with bounded resource usage.

### Exit Criteria
- Durable names are deterministic and collision-free across replicas.
- Deliver groups are used where appropriate to share work across replicas.
- Exactly-once side effects are enforced via idempotency + dedupe keys (not wishful thinking).
- A scale-out test suite exists and is gated but runnable.

### Tasks
- [ ] Establish consumer naming scheme per service role:
  - [ ] Projection: per-view durable option uses sanitized names and stable mapping
  - [ ] Runner: durable prefix includes role + shard + optional group
- [ ] Establish deliver group usage rules:
  - [ ] when to enable (scale-out consumers)
  - [ ] how to roll without duplication
- [ ] Strengthen dedupe keys:
  - [ ] event-driven sagas: checkpoint + dedupe marker strategy tested under redelivery
  - [ ] outbox relay: verify publish idempotency with `Nats-Msg-Id`
- [ ] Add gated tests:
  - [ ] two replicas, same tenant, no duplicate publishes
  - [ ] rolling restart preserves checkpoint correctness

## Verification Commands (Required at Each Milestone)
- `cargo fmt --check`
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test --workspace`
- Gated NATS integration tests:
  - Runner: `RUNNER_TEST_NATS_URL=... cargo test -p runner -- --ignored`
  - Projection: `PROJECTION_TEST_NATS_URL=... cargo test -p projection -- --ignored`
  - Control API (if it runs NATS-gated tests): set documented env flags and run ignored tests

## Notes / Constraints
- Do not create per-tenant streams unless scaling evidence requires it; prefer subject partitioning and consumer groups.
- Prefer backward-compatible envelope changes (optional fields, tolerant decoding).
- Prefer stable durable consumers; ephemeral consumers must be unique and bounded and must cleanup best-effort.