added initial roadmap and implementation
This commit is contained in:
317
SPECIFICATIONS.md
Normal file
317
SPECIFICATIONS.md
Normal file
@@ -0,0 +1,317 @@
|
||||
### MadBase (Supabase-Compatible Rust API Layer) — Functional & Non-Functional Specification (Updated)
|
||||
|
||||
This document specifies **MadBase**, a **Supabase API-compatible** platform implemented primarily in **Rust**, designed to be a drop-in replacement for Supabase’s hosted API surface while supporting **Bring Your Own PostgreSQL**. Primary goals: **low resource usage**, **simplicity**, and **operability** (Docker Compose-first).
|
||||
|
||||
## 0. Key Decisions (Chosen for Resource Usage, Simplicity, Operability)
|
||||
|
||||
1. **Single Rust “gateway” binary** (one container) exposes Supabase-compatible endpoints:
|
||||
- `/auth/v1/*`, `/rest/v1/*`, `/rpc/v1/*` (or `/rest/v1/rpc/*` depending on compatibility requirements), `/storage/v1/*`, `/realtime/v1/*` (WS), `/functions/v1/*`, `/admin/v1/*`
|
||||
2. **Custom “PostgREST-lite” Rust data API** (instead of embedding PostgREST) to reduce runtime overhead and simplify deployment.
|
||||
3. **Realtime via PostgreSQL logical replication (pgoutput)** implemented in Rust (no external Debezium/Elixir stack).
|
||||
4. **Edge Functions executed as WASM via Wasmtime** (small, fast startup, strong sandbox). (Optionally support “Deno compatibility mode” later if required.)
|
||||
5. **Storage API is an S3 proxy** with MinIO default in podman-compose and support for external S3/R2/etc. in production.
|
||||
6. **Monitoring: VictoriaMetrics + Loki + Grafana** with Prometheus-format metrics endpoints and JSON stdout logs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Functional Requirements
|
||||
|
||||
### 1.1 Supabase Client Compatibility (Hard Requirement)
|
||||
**Goal:** `@supabase/supabase-js` works unchanged. Users only swap the URL.
|
||||
|
||||
#### 1.1.1 API Surface Compatibility
|
||||
MadBase must mimic Supabase’s public endpoints, including:
|
||||
- Path structure (e.g., `/auth/v1`, `/rest/v1`, `/storage/v1`, `/realtime/v1`, `/functions/v1`)
|
||||
- Required headers:
|
||||
- `apikey: <anon/service_role key>`
|
||||
- `Authorization: Bearer <jwt>`
|
||||
- Status codes and error format consistency (enough to not break supabase-js).
|
||||
|
||||
#### 1.1.2 Key Types
|
||||
- **Anon key**: public, used by clients for most operations.
|
||||
- **Service role key**: privileged, bypasses RLS (admin operations).
|
||||
- Key rotation supported per project.
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Multi-Tenant Projects (Control Plane)
|
||||
MadBase is multi-tenant and supports many “projects,” each mapping to:
|
||||
- A PostgreSQL connection string (BYO DB/cluster)
|
||||
- JWT secret / signing policy
|
||||
- Storage backend configuration
|
||||
- Function runtime configuration
|
||||
- Realtime replication configuration
|
||||
|
||||
#### 1.2.1 Project Identification
|
||||
Projects are identified via one of:
|
||||
- Subdomain-based routing: `https://<project-ref>.<base-domain>`
|
||||
- Or header-based routing: `x-project-ref: <ref>` (for internal/admin use)
|
||||
|
||||
#### 1.2.2 Project Lifecycle
|
||||
- Create project
|
||||
- Update project settings (DB URL, keys, storage config)
|
||||
- Disable / delete project (with soft-delete option)
|
||||
- View usage/health info per project
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Auth (GoTrue-Compatible-ish)
|
||||
MadBase provides auth flows expected by supabase-js.
|
||||
|
||||
#### 1.3.1 Supported Flows
|
||||
- Email/password signup
|
||||
- Email/password login
|
||||
- Token refresh (refresh token → new access token)
|
||||
- Session retrieval (`/user`)
|
||||
- Password reset (email-based) — optional for MVP, but spec’d
|
||||
- Email confirmation — optional for MVP, but spec’d
|
||||
- OAuth providers — **out of MVP**, future extension
|
||||
|
||||
#### 1.3.2 User Model
|
||||
For each project:
|
||||
- Users are isolated by `project_id`
|
||||
- Store:
|
||||
- `id (uuid)`
|
||||
- `email`
|
||||
- `password_hash (argon2)`
|
||||
- `created_at`
|
||||
- `confirmed_at` (optional)
|
||||
- `user_metadata` (jsonb)
|
||||
- `app_metadata` (jsonb, e.g., provider)
|
||||
|
||||
#### 1.3.3 Token Model
|
||||
- Access token: JWT, short-lived
|
||||
- Refresh token: opaque random string (stored hashed) or JWT-like; must support rotation
|
||||
- Required JWT claims to align with RLS expectations:
|
||||
- `sub`
|
||||
- `role` (e.g., `authenticated` / `anon`)
|
||||
- `aud`
|
||||
- `exp`
|
||||
- plus project scoping claim (e.g., `project_ref`)
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Data API (CRUD, Filtering, Ordering, Pagination)
|
||||
MadBase provides PostgREST-like behavior.
|
||||
|
||||
#### 1.4.1 CRUD
|
||||
- `GET /rest/v1/<table>`
|
||||
- `POST /rest/v1/<table>`
|
||||
- `PATCH /rest/v1/<table>`
|
||||
- `DELETE /rest/v1/<table>`
|
||||
|
||||
#### 1.4.2 Query Features
|
||||
- `select=*` and nested selects (MVP: basic selects; advanced nested relations phased)
|
||||
- Filters:
|
||||
- `eq`, `neq`, `lt`, `lte`, `gt`, `gte`
|
||||
- `like`, `ilike`
|
||||
- `in`
|
||||
- `is` (null checks)
|
||||
- `or` boolean expression support (MVP: limited; expand later)
|
||||
- Ordering: `order=col.asc|desc`
|
||||
- Pagination:
|
||||
- `limit`, `offset`
|
||||
- Range headers compatibility (nice-to-have if required by supabase-js usage patterns)
|
||||
- Count support (`Prefer: count=exact|planned|estimated`) as feasible
|
||||
|
||||
#### 1.4.3 RPC
|
||||
- Postgres function invocation endpoint compatible with Supabase usage patterns:
|
||||
- `POST /rest/v1/rpc/<function>` (common supabase pattern)
|
||||
- Input via JSON body; output as JSON.
|
||||
|
||||
#### 1.4.4 RLS Enforcement (Hard Requirement)
|
||||
MadBase must rely on **native PostgreSQL RLS**.
|
||||
|
||||
For each request within a transaction:
|
||||
- Validate JWT (or anon)
|
||||
- Set request-local variables:
|
||||
- `SET LOCAL request.jwt.claims = '<json>'` OR granular `SET LOCAL request.jwt.claim.<x>`
|
||||
- Set role appropriately:
|
||||
- anon → restricted
|
||||
- authenticated → restricted
|
||||
- service role → bypass policies (by using privileged DB role / bypass behavior)
|
||||
|
||||
**Outcome:** Existing Supabase-style RLS policies work unchanged.
|
||||
|
||||
---
|
||||
|
||||
### 1.5 Realtime (WebSocket Subscriptions)
|
||||
MadBase supports Supabase-js subscriptions.
|
||||
|
||||
#### 1.5.1 WebSocket Protocol Compatibility
|
||||
- `supabase.channel(...).on('postgres_changes', ...)` works unchanged (protocol-level compatibility target)
|
||||
- Support channel join/leave, heartbeats, and authorization.
|
||||
|
||||
#### 1.5.2 Change Capture
|
||||
- Use PostgreSQL logical replication (`pgoutput`)
|
||||
- Per-project replication slot management
|
||||
- Resume from last confirmed LSN after restart
|
||||
|
||||
#### 1.5.3 Filtering + RLS Semantics
|
||||
- Enforce subscription filters (table, schema, event types, optional column filters)
|
||||
- **RLS-correctness goal**: only stream rows the user is allowed to see.
|
||||
- MVP approach: re-check row visibility by executing a parameterized query under the user’s claims (heavier but correct)
|
||||
- Future: optimize with precomputed policies / projection strategies
|
||||
|
||||
---
|
||||
|
||||
### 1.6 Object Storage (Supabase Storage-Compatible)
|
||||
MadBase provides:
|
||||
- Bucket CRUD
|
||||
- Object upload/download/delete
|
||||
- Signed URL generation (optional)
|
||||
- Public/private bucket behavior
|
||||
|
||||
#### 1.6.1 Storage Backend
|
||||
- Default: MinIO in Podman Compose
|
||||
- Production: external S3-compatible endpoint
|
||||
- Metadata stored per project in control DB (or optionally tenant DB)
|
||||
|
||||
#### 1.6.2 Authorization
|
||||
- Requires JWT or anon access depending on bucket policy
|
||||
- Service role bypass supported
|
||||
|
||||
---
|
||||
|
||||
### 1.7 Edge Functions (Full Support, WASM Runtime)
|
||||
MadBase supports:
|
||||
- `POST /functions/v1/<name>` invocation
|
||||
- Deploy/update function artifacts per project
|
||||
|
||||
#### 1.7.1 Runtime
|
||||
- Wasmtime sandbox execution
|
||||
- Per-invocation limits:
|
||||
- max CPU time
|
||||
- max memory
|
||||
- max request body size
|
||||
- Environment variables injection:
|
||||
- project ref
|
||||
- supabase URL
|
||||
- anon key (optional)
|
||||
- secrets (encrypted at rest)
|
||||
|
||||
#### 1.7.2 Function Packaging
|
||||
- MVP: accept WASM modules + manifest
|
||||
- Optional toolchain support for TS/JS→WASM documented, but not required in-core.
|
||||
|
||||
---
|
||||
|
||||
### 1.8 Management UI (Full Project Management)
|
||||
UI provides:
|
||||
- Project creation/config
|
||||
- DB connection configuration test
|
||||
- Key management + rotation
|
||||
- User management (admin)
|
||||
- Storage bucket management
|
||||
- Edge function deployment + logs
|
||||
- Realtime connection stats
|
||||
- RLS policy helpers (optional, best-effort)
|
||||
- Health checks
|
||||
|
||||
UI talks to `/admin/v1/*` endpoints.
|
||||
|
||||
---
|
||||
|
||||
### 1.9 Podman Compose Deployment
|
||||
Single `docker-compose.yml` (compatible with `podman-compose`) deploys:
|
||||
- MadBase API container
|
||||
- Control plane Postgres (for project/user/config)
|
||||
- MinIO (default storage)
|
||||
- Grafana
|
||||
- Loki
|
||||
- VictoriaMetrics
|
||||
|
||||
BYO tenant Postgres DBs are external (not necessarily in compose).
|
||||
|
||||
---
|
||||
|
||||
## 2. Non-Functional Requirements
|
||||
|
||||
### 2.1 Resource Usage Targets
|
||||
- **Single-node dev deployment** (compose):
|
||||
- MadBase (idle) memory target: ~100–300MB depending on active WS connections
|
||||
- Minimal CPU when idle
|
||||
- Monitoring stack runs within reasonable bounds for small installs.
|
||||
|
||||
### 2.2 Performance
|
||||
- REST overhead target: low single-digit milliseconds excluding DB time
|
||||
- WebSocket fanout efficiency:
|
||||
- handle thousands to tens of thousands connections per node (depending on hardware)
|
||||
- Streaming uploads/downloads without buffering entire objects in memory.
|
||||
|
||||
### 2.3 Operability (Primary Goal)
|
||||
- **Few moving parts**: one Rust service container + standard off-the-shelf observability components.
|
||||
- Stateless API layer:
|
||||
- can scale horizontally behind a load balancer
|
||||
- Configuration via environment variables + control DB records.
|
||||
- Smooth upgrades:
|
||||
- migrations for control plane DB
|
||||
- backward compatible API where possible
|
||||
|
||||
### 2.4 Reliability & Availability
|
||||
- Graceful shutdown:
|
||||
- drain in-flight requests
|
||||
- close WS connections cleanly
|
||||
- Realtime resume:
|
||||
- store last confirmed LSN per project
|
||||
- Retry policies for transient DB/storage failures
|
||||
- Backpressure handling for WS broadcasts and replication consumption
|
||||
|
||||
### 2.5 Security
|
||||
- JWT validation strictness (issuer/audience configurable per project)
|
||||
- Password hashing: Argon2
|
||||
- Secrets encrypted at rest in control DB
|
||||
- RBAC for admin UI
|
||||
- Edge runtime sandboxing:
|
||||
- no filesystem by default
|
||||
- no network by default (or allowlist)
|
||||
- Defense-in-depth:
|
||||
- rate limiting per project/IP
|
||||
- request size limits
|
||||
- CORS configuration
|
||||
|
||||
### 2.6 Observability
|
||||
- Metrics:
|
||||
- request counts/latency by endpoint and project
|
||||
- DB pool stats
|
||||
- replication lag
|
||||
- WS connections
|
||||
- function invocation duration/errors
|
||||
- storage bytes in/out
|
||||
- Logs:
|
||||
- JSON structured logs to stdout
|
||||
- correlation IDs per request
|
||||
- Dashboards:
|
||||
- Grafana dashboards included for the above
|
||||
|
||||
### 2.7 Maintainability
|
||||
- Modular internal crate structure:
|
||||
- `gateway`, `auth`, `data_api`, `realtime`, `storage`, `functions`, `admin`, `control_plane`
|
||||
- Clear compatibility test suite using supabase-js integration tests
|
||||
- Versioned API surface for `/admin/v1`
|
||||
|
||||
---
|
||||
|
||||
## 3. Constraints & Assumptions
|
||||
|
||||
- MadBase is **API layer only**; it does not provision tenant databases (though it can optionally assist with templates).
|
||||
- PostgreSQL is assumed to be standard and supports:
|
||||
- RLS
|
||||
- logical replication (for realtime)
|
||||
- Supabase-js compatibility is the north star; where full fidelity is expensive:
|
||||
- MVP supports the most common subset, with explicit “compatibility gaps” tracked.
|
||||
|
||||
---
|
||||
|
||||
## 4. Acceptance Criteria (Definition of Done)
|
||||
|
||||
1. A standard supabase-js app can:
|
||||
- sign up, log in, refresh tokens
|
||||
- perform CRUD with filters/order/pagination
|
||||
- call RPC functions
|
||||
- upload/download storage objects
|
||||
- receive realtime row change events
|
||||
- invoke an edge function
|
||||
2. Two projects pointing at two different external Postgres instances are fully isolated.
|
||||
3. RLS policies behave identically to Supabase for supported query patterns.
|
||||
4. Entire stack deploys via podman-compose and exposes Grafana dashboards with logs+metrics.
|
||||
Reference in New Issue
Block a user