madbase/SPECIFICATIONS.md

MadBase (Supabase-Compatible Rust API Layer) — Functional & Non-Functional Specification (Updated)

This document specifies MadBase, a Supabase API-compatible platform implemented primarily in Rust, designed to be a drop-in replacement for Supabase’s hosted API surface while supporting Bring Your Own PostgreSQL. Primary goals: low resource usage, simplicity, and operability (Docker Compose-first).

0. Key Decisions (Chosen for Resource Usage, Simplicity, Operability)

1. Single Rust “gateway” binary (one container) exposes Supabase-compatible endpoints:
   • /auth/v1/*, /rest/v1/*, /rpc/v1/* (or /rest/v1/rpc/* depending on compatibility requirements), /storage/v1/*, /realtime/v1/* (WS), /functions/v1/*, /admin/v1/*
2. Custom “PostgREST-lite” Rust data API (instead of embedding PostgREST) to reduce runtime overhead and simplify deployment.
3. Realtime via PostgreSQL logical replication (pgoutput) implemented in Rust (no external Debezium/Elixir stack).
4. Edge Functions executed as WASM via Wasmtime (small, fast startup, strong sandbox). (Optionally support “Deno compatibility mode” later if required.)
5. Storage API is an S3 proxy with MinIO default in podman-compose and support for external S3/R2/etc. in production.
6. Monitoring: VictoriaMetrics + Loki + Grafana with Prometheus-format metrics endpoints and JSON stdout logs.

---

1. Functional Requirements

1.1 Supabase Client Compatibility (Hard Requirement)

Goal: @supabase/supabase-js works unchanged. Users only swap the URL.

1.1.1 API Surface Compatibility

MadBase must mimic Supabase’s public endpoints, including:

• Path structure (e.g., /auth/v1, /rest/v1, /storage/v1, /realtime/v1, /functions/v1)
• Required headers:
  • apikey: <anon/service_role key>
  • Authorization: Bearer <jwt>
• Status codes and error format consistency (enough to not break supabase-js).

1.1.2 Key Types

• Anon key: public, used by clients for most operations.
• Service role key: privileged, bypasses RLS (admin operations).
• Key rotation supported per project.

---

1.2 Multi-Tenant Projects (Control Plane)

MadBase is multi-tenant and supports many “projects,” each mapping to:

• A PostgreSQL connection string (BYO DB/cluster)
• JWT secret / signing policy
• Storage backend configuration
• Function runtime configuration
• Realtime replication configuration

1.2.1 Project Identification

Projects are identified via one of:

• Subdomain-based routing: https://<project-ref>.<base-domain>
• Or header-based routing: x-project-ref: <ref> (for internal/admin use)

1.2.2 Project Lifecycle

• Create project
• Update project settings (DB URL, keys, storage config)
• Disable / delete project (with soft-delete option)
• View usage/health info per project

---

1.3 Auth (GoTrue-Compatible-ish)

MadBase provides auth flows expected by supabase-js.

1.3.1 Supported Flows

• Email/password signup
• Email/password login
• Token refresh (refresh token → new access token)
• Session retrieval (/user)
• Password reset (email-based) — optional for MVP, but spec’d
• Email confirmation — optional for MVP, but spec’d
• OAuth providers — out of MVP, future extension

1.3.2 User Model

For each project:

• Users are isolated by project_id
• Store:
  • id (uuid)
  • email
  • password_hash (argon2)
  • created_at
  • confirmed_at (optional)
  • user_metadata (jsonb)
  • app_metadata (jsonb, e.g., provider)

1.3.3 Token Model

• Access token: JWT, short-lived
• Refresh token: opaque random string (stored hashed) or JWT-like; must support rotation
• Required JWT claims to align with RLS expectations:
  • sub
  • role (e.g., authenticated / anon)
  • aud
  • exp
  • plus project scoping claim (e.g., project_ref)

---

1.4 Data API (CRUD, Filtering, Ordering, Pagination)

MadBase provides PostgREST-like behavior.

1.4.1 CRUD

• GET /rest/v1/<table>
• POST /rest/v1/<table>
• PATCH /rest/v1/<table>
• DELETE /rest/v1/<table>

1.4.2 Query Features

• select=* and nested selects (MVP: basic selects; advanced nested relations phased)
• Filters:
  • eq, neq, lt, lte, gt, gte
  • like, ilike
  • in
  • is (null checks)
  • or boolean expression support (MVP: limited; expand later)
• Ordering: order=col.asc|desc
• Pagination:
  • limit, offset
  • Range headers compatibility (nice-to-have if required by supabase-js usage patterns)
• Count support (Prefer: count=exact|planned|estimated) as feasible

1.4.3 RPC

• Postgres function invocation endpoint compatible with Supabase usage patterns:
  • POST /rest/v1/rpc/<function> (common supabase pattern)
• Input via JSON body; output as JSON.

1.4.4 RLS Enforcement (Hard Requirement)

MadBase must rely on native PostgreSQL RLS.

For each request within a transaction:

• Validate JWT (or anon)
• Set request-local variables:
  • SET LOCAL request.jwt.claims = '<json>' OR granular SET LOCAL request.jwt.claim.<x>
• Set role appropriately:
  • anon → restricted
  • authenticated → restricted
  • service role → bypass policies (by using privileged DB role / bypass behavior)

Outcome: Existing Supabase-style RLS policies work unchanged.

---

1.5 Realtime (WebSocket Subscriptions)

MadBase supports Supabase-js subscriptions.

1.5.1 WebSocket Protocol Compatibility

• supabase.channel(...).on('postgres_changes', ...) works unchanged (protocol-level compatibility target)
• Support channel join/leave, heartbeats, and authorization.

1.5.2 Change Capture

• Use PostgreSQL logical replication (pgoutput)
• Per-project replication slot management
• Resume from last confirmed LSN after restart

1.5.3 Filtering + RLS Semantics

• Enforce subscription filters (table, schema, event types, optional column filters)
• RLS-correctness goal: only stream rows the user is allowed to see.
  • MVP approach: re-check row visibility by executing a parameterized query under the user’s claims (heavier but correct)
  • Future: optimize with precomputed policies / projection strategies

---

1.6 Object Storage (Supabase Storage-Compatible)

MadBase provides:

• Bucket CRUD
• Object upload/download/delete
• Signed URL generation (optional)
• Public/private bucket behavior

1.6.1 Storage Backend

• Default: MinIO in Podman Compose
• Production: external S3-compatible endpoint
• Metadata stored per project in control DB (or optionally tenant DB)

1.6.2 Authorization

• Requires JWT or anon access depending on bucket policy
• Service role bypass supported

---

1.7 Edge Functions (Full Support, WASM Runtime)

MadBase supports:

• POST /functions/v1/<name> invocation
• Deploy/update function artifacts per project

1.7.1 Runtime

• Wasmtime sandbox execution
• Per-invocation limits:
  • max CPU time
  • max memory
  • max request body size
• Environment variables injection:
  • project ref
  • supabase URL
  • anon key (optional)
  • secrets (encrypted at rest)

1.7.2 Function Packaging

• MVP: accept WASM modules + manifest
• Optional toolchain support for TS/JS→WASM documented, but not required in-core.

---

1.8 Management UI (Full Project Management)

UI provides:

• Project creation/config
• DB connection configuration test
• Key management + rotation
• User management (admin)
• Storage bucket management
• Edge function deployment + logs
• Realtime connection stats
• RLS policy helpers (optional, best-effort)
• Health checks

UI talks to /admin/v1/* endpoints.

---

1.9 Podman Compose Deployment

Single docker-compose.yml (compatible with podman-compose) deploys:

• MadBase API container
• Control plane Postgres (for project/user/config)
• MinIO (default storage)
• Grafana
• Loki
• VictoriaMetrics

BYO tenant Postgres DBs are external (not necessarily in compose).

---

2. Non-Functional Requirements

2.1 Resource Usage Targets

• Single-node dev deployment (compose):
  • MadBase (idle) memory target: ~100–300MB depending on active WS connections
  • Minimal CPU when idle
• Monitoring stack runs within reasonable bounds for small installs.

2.2 Performance

• REST overhead target: low single-digit milliseconds excluding DB time
• WebSocket fanout efficiency:
  • handle thousands to tens of thousands connections per node (depending on hardware)
• Streaming uploads/downloads without buffering entire objects in memory.

2.3 Operability (Primary Goal)

• Few moving parts: one Rust service container + standard off-the-shelf observability components.
• Stateless API layer:
  • can scale horizontally behind a load balancer
• Configuration via environment variables + control DB records.
• Smooth upgrades:
  • migrations for control plane DB
  • backward compatible API where possible

2.4 Reliability & Availability

• Graceful shutdown:
  • drain in-flight requests
  • close WS connections cleanly
• Realtime resume:
  • store last confirmed LSN per project
• Retry policies for transient DB/storage failures
• Backpressure handling for WS broadcasts and replication consumption

2.5 Security

• JWT validation strictness (issuer/audience configurable per project)
• Password hashing: Argon2
• Secrets encrypted at rest in control DB
• RBAC for admin UI
• Edge runtime sandboxing:
  • no filesystem by default
  • no network by default (or allowlist)
• Defense-in-depth:
  • rate limiting per project/IP
  • request size limits
  • CORS configuration

2.6 Observability

• Metrics:
  • request counts/latency by endpoint and project
  • DB pool stats
  • replication lag
  • WS connections
  • function invocation duration/errors
  • storage bytes in/out
• Logs:
  • JSON structured logs to stdout
  • correlation IDs per request
• Dashboards:
  • Grafana dashboards included for the above

2.7 Maintainability

• Modular internal crate structure:
  • gateway, auth, data_api, realtime, storage, functions, admin, control_plane
• Clear compatibility test suite using supabase-js integration tests
• Versioned API surface for /admin/v1

---

3. Constraints & Assumptions

• MadBase is API layer only; it does not provision tenant databases (though it can optionally assist with templates).
• PostgreSQL is assumed to be standard and supports:
  • RLS
  • logical replication (for realtime)
• Supabase-js compatibility is the north star; where full fidelity is expensive:
  • MVP supports the most common subset, with explicit “compatibility gaps” tracked.

---

4. Acceptance Criteria (Definition of Done)

1. A standard supabase-js app can:
   • sign up, log in, refresh tokens
   • perform CRUD with filters/order/pagination
   • call RPC functions
   • upload/download storage objects
   • receive realtime row change events
   • invoke an edge function
2. Two projects pointing at two different external Postgres instances are fully isolated.
3. RLS policies behave identically to Supabase for supported query patterns.
4. Entire stack deploys via podman-compose and exposes Grafana dashboards with logs+metrics.