# S3-Compatible Object Storage Plan (Hetzner in Prod, MinIO Locally)

## Principles
- S3-compatible object storage is mandatory for platform document storage in every environment:
  - Local development uses MinIO.
  - Production uses Hetzner Object Storage (S3 API compatible).
- Each milestone is stop-the-line gated:
  - All tasks completed
  - All milestone tests pass
  - Workspace verification commands pass
- Secrets are never committed and never logged:
  - Access keys via Swarm secrets in production
  - `.env` or compose env in local dev

## Goals
- Introduce a single, shared S3-compatible configuration surface for the platform.
- Make document storage always backed by S3 (no filesystem fallback for documents).
- Keep the implementation incremental and test-gated per milestone.
- Optionally expand to observability object storage after document storage is stable.

## Definitions
### Document Storage
“Documents” are versioned blobs the platform needs to store and retrieve reliably:
- Deployment bundles and artifacts
- Definitions/manifests (projection programs, saga/effects definitions, schema bundles)
- Exported audit/log bundles, diagnostics, or snapshots that are not part of the primary KV/MDBX state

Document storage must support:
- Tenant-scoped namespaces (prefixes)
- Content-addressed or versioned keys (immutability preferred)
- Listing by prefix for admin workflows

## Configuration Contract (Platform-Wide)
### Common Settings
- `S3_ENDPOINT` (Hetzner: HTTPS endpoint; MinIO: `http://minio:9000`)
- `S3_REGION` (required even for some S3-compatible providers)
- `S3_ACCESS_KEY_ID` (secret)
- `S3_SECRET_ACCESS_KEY` (secret)
- `S3_FORCE_PATH_STYLE` (`true/false`)
- `S3_INSECURE` (`true/false`, only allowed for local MinIO)

### Buckets and Prefixes
- `S3_BUCKET_DOCS` (required everywhere)
- `S3_PREFIX_DOCS` (default `docs/`)

Optional (later milestones):
- `S3_BUCKET_LOKI`, `S3_PREFIX_LOKI`
- `S3_BUCKET_TEMPO`, `S3_PREFIX_TEMPO`

## Target Architecture
### Local Development
- MinIO is part of the local stack for parity.
- Control API is the document gateway:
  - Upload/download via signed URLs or streamed proxy endpoints
  - Metadata stored in existing storage/KV (document index) or derived from key scheme

### Production
- Hetzner Object Storage provides S3-compatible bucket(s).
- Credentials and bucket details injected via Swarm secrets and stack env.

## Development Plan (Milestones by Dependency)

## Milestone 0: S3 Contract + Local MinIO Baseline
### Dependencies
- None

### Goal
Provide a consistent local S3-compatible endpoint and stable bucket naming to unblock higher milestones.

### Tasks
- [ ] Add MinIO to local development stack:
  - [ ] Add `minio` service to compose (API + console)
  - [ ] Add `minio-init` job to create required buckets
- [ ] Define standard bucket/prefix defaults for local dev:
  - [ ] `S3_BUCKET_DOCS=cloudlysis-docs`
  - [ ] `S3_PREFIX_DOCS=docs/`
- [ ] Document local workflow to enable MinIO-backed document storage.

### Required Tests (Gate)
- [ ] Workspace verification commands
- [ ] Local manual verification checklist:
  - [ ] `cloudlysis-docs` bucket exists
  - [ ] credentials work from a container in the compose network

## Milestone 1: Document Storage API (Control API)
### Dependencies
- Milestone 0

### Goal
Make document storage a first-class platform API and require it in all environments.

### Tasks
- [ ] Add an S3 client module to Control API:
  - [ ] parse config from env with strict validation (endpoint, bucket, keys)
  - [ ] support path-style and TLS/insecure options
- [ ] Implement document primitives:
  - [ ] Put (upload) and Get (download)
  - [ ] List by prefix (tenant + doc-type)
  - [ ] Delete (admin-only) if needed
- [ ] Decide and document a key scheme:
  - [ ] tenant-scoped prefix
  - [ ] immutable keys preferred (content hash + metadata)
- [ ] Add authz rules for document operations (deny-by-default, tenant-scoped).

### Required Tests (Gate)
- [ ] Workspace verification commands
- [ ] Unit tests:
  - [ ] config parsing/validation
  - [ ] key generation stability
- [ ] Gated integration tests (MinIO):
  - [ ] put/get roundtrip
  - [ ] list by prefix
  - [ ] tenant isolation (cannot read other tenant prefix)

## Milestone 2: Control UI Integration (Upload/Download Flows)
### Dependencies
- Milestone 1

### Goal
Make document workflows usable from the Control UI without leaking credentials.

### Tasks
- [ ] Add Control API endpoints for signed URLs (recommended) or streamed proxy:
  - [ ] create upload URL (PUT)
  - [ ] create download URL (GET)
- [ ] Implement Control UI flows for a first document type:
  - [ ] upload
  - [ ] list
  - [ ] download
- [ ] Ensure correlation/trace propagation on Control API operations.

### Required Tests (Gate)
- [ ] Workspace verification commands
- [ ] Control UI unit tests for routing/component render stability
- [ ] Gated end-to-end checklist (local):
  - [ ] upload appears in list
  - [ ] download returns expected bytes

## Milestone 3: Production Rollout (Hetzner)
### Dependencies
- Milestone 2

### Goal
Deploy document storage on Hetzner S3-compatible backend with production-grade secret handling.

### Tasks
- [ ] Provision buckets and lifecycle policies (docs bucket):
  - [ ] retention rules appropriate to documents
  - [ ] access policy scoped to required actions
- [ ] Swarm deployment:
  - [ ] add secrets for access keys
  - [ ] configure Control API with endpoint/region/bucket/prefix
- [ ] Rollback plan:
  - [ ] switch to a fallback bucket or MinIO-on-prod if needed

### Required Tests (Gate)
- [ ] Workspace verification commands
- [ ] Production smoke runbook:
  - [ ] upload/list/download for a tenant
  - [ ] verify objects exist under expected prefixes

## Milestone 4 (Optional): Observability Storage on S3 (Loki + Tempo)
### Dependencies
- Milestone 3

### Goal
Store logs and traces in S3-compatible storage (MinIO locally; Hetzner in production).

### Tasks
- [ ] Loki:
  - [ ] add S3 config variant and compose overlay
  - [ ] validate log query and bucket objects
- [ ] Tempo:
  - [ ] add S3 config variant and compose overlay
  - [ ] validate traces and bucket objects

### Required Tests (Gate)
- [ ] Workspace verification commands
- [ ] Gated local validation:
  - [ ] Loki writes objects to bucket/prefix after ingest
  - [ ] Tempo writes objects to bucket/prefix after ingest

## Workspace Verification Commands
- `cargo fmt --check`
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test --workspace`
- `cd control/ui && npm ci && npm run lint && npm run typecheck && npm run test && npm run build`