madbase/_milestones/M4_data_api.md

# Milestone 4: Data API Completeness

**Goal:** `supabase.from(table).select().eq().order()` and the full PostgREST query surface works.

**Depends on:** M0 (Security), M1 (Foundation)

---

## 4.1 — Missing Operators & Features

### Implementation approach

All operators are parsed in `data_api/src/parser.rs` and applied in `data_api/src/handlers.rs`. The parser already handles `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `like`, `ilike`, `in`, `is`. It also has partial `or`/`and` support in `FilterNode::parse`.

### 4.1.1 or / not filters

The parser already parses `or(col1.eq.a,col2.eq.b)` into `FilterNode::Or(...)`. Verify the SQL generation in `build_filter_clause` correctly emits `(col1 = 'a' OR col2 = 'b')`.

Add `not` operator:
```rust
// In parser.rs Operator enum
Not,  // Wraps another condition with NOT

// In parser.rs
"not" => Some(Operator::Not),

// In to_sql
Operator::Not => "NOT",
```

Usage: `?name=not.eq.null` → `NOT (name = NULL)` or more correctly `name IS NOT NULL`.

### 4.1.2 contains / containedBy

For JSONB and array columns:

```rust
Operator::Contains => "@>",
Operator::ContainedBy => "<@",
```

Parse: `?tags=cs.{a,b}` → `tags @> ARRAY['a','b']`

### 4.1.3 textSearch

```rust
Operator::TextSearch => "@@",
```

Parse: `?content=fts.hello+world` → `to_tsvector(content) @@ plainto_tsquery('hello world')`

### 4.1.4 Range pagination

Read `Range` header in handler:
```rust
let range = headers.get("Range")
    .and_then(|v| v.to_str().ok())
    .and_then(|s| {
        let parts: Vec<&str> = s.split('-').collect();
        Some((parts[0].parse::<usize>().ok()?, parts[1].parse::<usize>().ok()?))
    });

if let Some((start, end)) = range {
    // Add OFFSET start LIMIT (end - start + 1)
    // Set Content-Range header in response: "0-9/100"
}
```

### 4.1.5 Prefer: count=exact

Read `Prefer` header:
```rust
let want_count = headers.get("Prefer")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("count=exact"))
    .unwrap_or(false);

if want_count {
    // Run a parallel COUNT(*) query
    // Set Content-Range: "0-9/42" or "*/42"
}
```

### 4.1.6 single / maybeSingle

Read `Accept` header:
```rust
let want_single = headers.get("Accept")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("vnd.pgrst.object+json"))
    .unwrap_or(false);

if want_single {
    // LIMIT 1, return object instead of array
    // If no rows: 406 Not Acceptable (for single), null (for maybeSingle)
}
```

### 4.1.7 Upsert

Read `Prefer` header for `resolution=merge-duplicates`:
```rust
let prefer_upsert = headers.get("Prefer")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("resolution=merge-duplicates"))
    .unwrap_or(false);

if prefer_upsert {
    // INSERT ... ON CONFLICT DO UPDATE SET col1 = EXCLUDED.col1, ...
}
```

### 4.1.8 RPC support

**File:** `data_api/src/handlers.rs` (new), `data_api/src/lib.rs` (add route)

```rust
.route("/rpc/:function_name", post(handlers::call_rpc))
```

```rust
pub async fn call_rpc(
    State(state): State<DataState>,
    Extension(auth_ctx): Extension<AuthContext>,
    Path(function_name): Path<String>,
    Json(params): Json<serde_json::Value>,
    db: Option<Extension<PgPool>>,
) -> Result<Json<serde_json::Value>, ApiError> {
    // Validate function_name is a valid identifier
    if !is_valid_identifier(&function_name) {
        return Err(ApiError::BadRequest("Invalid function name".into()));
    }

    let pool = db.map(|Extension(p)| p).unwrap_or(state.db.clone());
    let mut rls = RlsTransaction::begin(&pool, &auth_ctx).await?;

    // Build: SELECT * FROM function_name($1)
    let query = format!("SELECT * FROM {}($1::jsonb)", function_name);
    let rows = sqlx::query(&query)
        .bind(&params)
        .fetch_all(&mut *rls.tx)
        .await?;

    let result = rows_to_json(rows);
    Ok(Json(result))
}
```

### 4.1.9 Schema selection

Read `Accept-Profile` / `Content-Profile` headers:
```rust
let schema = headers.get("Accept-Profile")
    .or(headers.get("Content-Profile"))
    .and_then(|v| v.to_str().ok())
    .unwrap_or("public");

// Validate schema exists
// Add SET LOCAL search_path = schema in the RLS transaction
```

---

## 4.2 — Nested Resource Embedding

This is the most complex feature. PostgREST's `select=*,author:users(*)` generates JOINs based on FK relationships.

### Phase 1: Single-level explicit FK

The parser already handles `SelectNode::Relation("author:users", inner_columns)` via `SelectNode::parse`. The handler needs to:

1. Detect `Relation` nodes in the select list
2. Look up the FK between the main table and the related table
3. Generate a LEFT JOIN or subquery
4. Nest the results in the JSON response

**Schema introspection query:**
```sql
SELECT
    tc.constraint_name,
    kcu.column_name AS fk_column,
    ccu.table_schema AS referenced_schema,
    ccu.table_name AS referenced_table,
    ccu.column_name AS referenced_column
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage ccu ON ccu.constraint_name = tc.constraint_name
WHERE tc.constraint_type = 'FOREIGN KEY' AND tc.table_name = $1
```

**Cache this** per table (see 4.3).

### Phase 2: Multi-level nesting

Recursive: for each `Relation` node, apply the same embedding logic to its inner `Relation` nodes.

### Phase 3: Computed/virtual relationships

Allow `!inner` (INNER JOIN) and `!left` (LEFT JOIN) hints in the select parameter.

---

## 4.3 — Performance

### 4.3.1 Cache schema introspection

Create a `SchemaCache` that loads FK and column metadata on first request per table, caches with 5-minute TTL:

```rust
use moka::future::Cache;

pub struct SchemaCache {
    fk_cache: Cache<String, Vec<ForeignKey>>,
    column_cache: Cache<String, Vec<ColumnInfo>>,
}

impl SchemaCache {
    pub fn new() -> Self {
        Self {
            fk_cache: Cache::builder().time_to_live(Duration::from_secs(300)).build(),
            column_cache: Cache::builder().time_to_live(Duration::from_secs(300)).build(),
        }
    }
}
```

Invalidate on DDL changes by listening to `pg_notify('schema_change', ...)` via a background task.

---

## Completion Requirements

This milestone is **not complete** until every item below is satisfied.

### 1. Full Test Suite — All Green

- [ ] `cargo test --workspace` passes with **zero failures**
- [ ] All **pre-existing tests** still pass (no regressions)
- [ ] **New unit tests** are written for every feature in this milestone:

| Test | Location | What it validates |
|------|----------|-------------------|
| `test_parse_or_filter` | `data_api/src/parser.rs` | `or(title.eq.A,title.eq.B)` generates correct SQL with `OR` |
| `test_parse_not_filter` | `data_api/src/parser.rs` | `not.status.eq.draft` generates `NOT (status = 'draft')` |
| `test_parse_contains_jsonb` | `data_api/src/parser.rs` | `tags.cs.{a,b}` generates `tags @> $1` |
| `test_parse_contained_by` | `data_api/src/parser.rs` | `tags.cd.{a,b,c}` generates `tags <@ $1` |
| `test_parse_text_search` | `data_api/src/parser.rs` | `fts.english.hello` generates `to_tsvector('english', col) @@ to_tsquery($1)` |
| `test_range_header_pagination` | `data_api/src/handlers.rs` | `Range: 0-9` returns 10 rows with `Content-Range: 0-9/*` |
| `test_count_exact_header` | `data_api/src/handlers.rs` | `Prefer: count=exact` returns `Content-Range: 0-N/TOTAL` |
| `test_single_object_response` | `data_api/src/handlers.rs` | `Accept: application/vnd.pgrst.object+json` returns a single JSON object, not array |
| `test_single_object_406_on_multiple` | `data_api/src/handlers.rs` | Single-object mode with 2+ rows returns 406 |
| `test_upsert_merge_duplicates` | `data_api/src/handlers.rs` | `Prefer: resolution=merge-duplicates` upserts correctly |
| `test_rpc_call` | `data_api/src/handlers.rs` | `POST /rest/v1/rpc/my_func` with JSON params calls the function and returns results |
| `test_rpc_invalid_name_rejected` | `data_api/src/handlers.rs` | `POST /rest/v1/rpc/drop table` returns 400 |
| `test_schema_selection` | `data_api/src/handlers.rs` | `Accept-Profile: custom_schema` queries the correct schema |
| `test_nested_select_fk_join` | `data_api/src/handlers.rs` | `select=*,author:users(name)` returns nested objects |
| `test_schema_cache_invalidation` | `data_api/src/handlers.rs` | Schema cache refreshes after DDL changes (or after TTL) |

### 2. Integration / supabase-js Compatibility Verification

- [ ] `supabase.from('posts').select('*').or('title.eq.Hello,title.eq.World')` returns matching rows
- [ ] `supabase.from('posts').select('*, author:users(name)')` returns nested author objects
- [ ] `supabase.from('posts').select('*', { count: 'exact' })` returns count in `Content-Range` header
- [ ] `supabase.from('posts').upsert({ id: 1, title: 'Updated' })` creates or updates
- [ ] `supabase.rpc('my_function', { param: 'value' })` calls the Postgres function
- [ ] `supabase.from('posts').select('*').range(0, 9)` returns first 10 rows with `Content-Range`
- [ ] Schema selection via `Accept-Profile` header works
- [ ] `.single()` returns one object (not array) and 406 on 0 or 2+ results
- [ ] `.maybeSingle()` returns one object or null

### 3. CI Gate

- [ ] All unit tests run in `cargo test --workspace`
- [ ] Parser tests are pure (no DB needed) and run on every PR
- [ ] Handler integration tests that require Postgres are documented and gated appropriately
- [ ] `cargo clippy --workspace -- -D warnings` passes with no new warnings