madbase/_milestones/M4_data_api.md

Milestone 4: Data API Completeness

Goal: supabase.from(table).select().eq().order() and the full PostgREST query surface works.

Depends on: M0 (Security), M1 (Foundation)

---

4.1 — Missing Operators & Features

Implementation approach

All operators are parsed in data_api/src/parser.rs and applied in data_api/src/handlers.rs. The parser already handles eq, neq, gt, gte, lt, lte, like, ilike, in, is. It also has partial or/and support in FilterNode::parse.

4.1.1 or / not filters

The parser already parses or(col1.eq.a,col2.eq.b) into FilterNode::Or(...). Verify the SQL generation in build_filter_clause correctly emits (col1 = 'a' OR col2 = 'b').

Add not operator:

// In parser.rs Operator enum
Not,  // Wraps another condition with NOT

// In parser.rs
"not" => Some(Operator::Not),

// In to_sql
Operator::Not => "NOT",

Usage: ?name=not.eq.null → NOT (name = NULL) or more correctly name IS NOT NULL.

4.1.2 contains / containedBy

For JSONB and array columns:

Operator::Contains => "@>",
Operator::ContainedBy => "<@",

Parse: ?tags=cs.{a,b} → tags @> ARRAY['a','b']

4.1.3 textSearch

Operator::TextSearch => "@@",

Parse: ?content=fts.hello+world → to_tsvector(content) @@ plainto_tsquery('hello world')

4.1.4 Range pagination

Read Range header in handler:

let range = headers.get("Range")
    .and_then(|v| v.to_str().ok())
    .and_then(|s| {
        let parts: Vec<&str> = s.split('-').collect();
        Some((parts[0].parse::<usize>().ok()?, parts[1].parse::<usize>().ok()?))
    });

if let Some((start, end)) = range {
    // Add OFFSET start LIMIT (end - start + 1)
    // Set Content-Range header in response: "0-9/100"
}

4.1.5 Prefer: count=exact

Read Prefer header:

let want_count = headers.get("Prefer")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("count=exact"))
    .unwrap_or(false);

if want_count {
    // Run a parallel COUNT(*) query
    // Set Content-Range: "0-9/42" or "*/42"
}

4.1.6 single / maybeSingle

Read Accept header:

let want_single = headers.get("Accept")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("vnd.pgrst.object+json"))
    .unwrap_or(false);

if want_single {
    // LIMIT 1, return object instead of array
    // If no rows: 406 Not Acceptable (for single), null (for maybeSingle)
}

4.1.7 Upsert

Read Prefer header for resolution=merge-duplicates:

let prefer_upsert = headers.get("Prefer")
    .and_then(|v| v.to_str().ok())
    .map(|s| s.contains("resolution=merge-duplicates"))
    .unwrap_or(false);

if prefer_upsert {
    // INSERT ... ON CONFLICT DO UPDATE SET col1 = EXCLUDED.col1, ...
}

4.1.8 RPC support

File: data_api/src/handlers.rs (new), data_api/src/lib.rs (add route)

.route("/rpc/:function_name", post(handlers::call_rpc))

pub async fn call_rpc(
    State(state): State<DataState>,
    Extension(auth_ctx): Extension<AuthContext>,
    Path(function_name): Path<String>,
    Json(params): Json<serde_json::Value>,
    db: Option<Extension<PgPool>>,
) -> Result<Json<serde_json::Value>, ApiError> {
    // Validate function_name is a valid identifier
    if !is_valid_identifier(&function_name) {
        return Err(ApiError::BadRequest("Invalid function name".into()));
    }

    let pool = db.map(|Extension(p)| p).unwrap_or(state.db.clone());
    let mut rls = RlsTransaction::begin(&pool, &auth_ctx).await?;

    // Build: SELECT * FROM function_name($1)
    let query = format!("SELECT * FROM {}($1::jsonb)", function_name);
    let rows = sqlx::query(&query)
        .bind(&params)
        .fetch_all(&mut *rls.tx)
        .await?;

    let result = rows_to_json(rows);
    Ok(Json(result))
}

4.1.9 Schema selection

Read Accept-Profile / Content-Profile headers:

let schema = headers.get("Accept-Profile")
    .or(headers.get("Content-Profile"))
    .and_then(|v| v.to_str().ok())
    .unwrap_or("public");

// Validate schema exists
// Add SET LOCAL search_path = schema in the RLS transaction

---

4.2 — Nested Resource Embedding

This is the most complex feature. PostgREST's select=*,author:users(*) generates JOINs based on FK relationships.

Phase 1: Single-level explicit FK

The parser already handles SelectNode::Relation("author:users", inner_columns) via SelectNode::parse. The handler needs to:

1. Detect Relation nodes in the select list
2. Look up the FK between the main table and the related table
3. Generate a LEFT JOIN or subquery
4. Nest the results in the JSON response

Schema introspection query:

SELECT
    tc.constraint_name,
    kcu.column_name AS fk_column,
    ccu.table_schema AS referenced_schema,
    ccu.table_name AS referenced_table,
    ccu.column_name AS referenced_column
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage ccu ON ccu.constraint_name = tc.constraint_name
WHERE tc.constraint_type = 'FOREIGN KEY' AND tc.table_name = $1

Cache this per table (see 4.3).

Phase 2: Multi-level nesting

Recursive: for each Relation node, apply the same embedding logic to its inner Relation nodes.

Phase 3: Computed/virtual relationships

Allow !inner (INNER JOIN) and !left (LEFT JOIN) hints in the select parameter.

---

4.3 — Performance

4.3.1 Cache schema introspection

Create a SchemaCache that loads FK and column metadata on first request per table, caches with 5-minute TTL:

use moka::future::Cache;

pub struct SchemaCache {
    fk_cache: Cache<String, Vec<ForeignKey>>,
    column_cache: Cache<String, Vec<ColumnInfo>>,
}

impl SchemaCache {
    pub fn new() -> Self {
        Self {
            fk_cache: Cache::builder().time_to_live(Duration::from_secs(300)).build(),
            column_cache: Cache::builder().time_to_live(Duration::from_secs(300)).build(),
        }
    }
}

Invalidate on DDL changes by listening to pg_notify('schema_change', ...) via a background task.

---

Completion Requirements

This milestone is not complete until every item below is satisfied.

1. Full Test Suite — All Green

• cargo test --workspace passes with zero failures
• All pre-existing tests still pass (no regressions)
• New unit tests are written for every feature in this milestone:

Test	Location	What it validates
test_parse_or_filter	data_api/src/parser.rs	or(title.eq.A,title.eq.B) generates correct SQL with OR
test_parse_not_filter	data_api/src/parser.rs	not.status.eq.draft generates NOT (status = 'draft')
test_parse_contains_jsonb	data_api/src/parser.rs	tags.cs.{a,b} generates tags @> $1
test_parse_contained_by	data_api/src/parser.rs	tags.cd.{a,b,c} generates tags <@ $1
test_parse_text_search	data_api/src/parser.rs	fts.english.hello generates to_tsvector('english', col) @@ to_tsquery($1)
test_range_header_pagination	data_api/src/handlers.rs	Range: 0-9 returns 10 rows with Content-Range: 0-9/*
test_count_exact_header	data_api/src/handlers.rs	Prefer: count=exact returns Content-Range: 0-N/TOTAL
test_single_object_response	data_api/src/handlers.rs	Accept: application/vnd.pgrst.object+json returns a single JSON object, not array
test_single_object_406_on_multiple	data_api/src/handlers.rs	Single-object mode with 2+ rows returns 406
test_upsert_merge_duplicates	data_api/src/handlers.rs	Prefer: resolution=merge-duplicates upserts correctly
test_rpc_call	data_api/src/handlers.rs	POST /rest/v1/rpc/my_func with JSON params calls the function and returns results
test_rpc_invalid_name_rejected	data_api/src/handlers.rs	POST /rest/v1/rpc/drop table returns 400
test_schema_selection	data_api/src/handlers.rs	Accept-Profile: custom_schema queries the correct schema
test_nested_select_fk_join	data_api/src/handlers.rs	select=*,author:users(name) returns nested objects
test_schema_cache_invalidation	data_api/src/handlers.rs	Schema cache refreshes after DDL changes (or after TTL)

2. Integration / supabase-js Compatibility Verification

• supabase.from('posts').select('*').or('title.eq.Hello,title.eq.World') returns matching rows
• supabase.from('posts').select('*, author:users(name)') returns nested author objects
• supabase.from('posts').select('*', { count: 'exact' }) returns count in Content-Range header
• supabase.from('posts').upsert({ id: 1, title: 'Updated' }) creates or updates
• supabase.rpc('my_function', { param: 'value' }) calls the Postgres function
• supabase.from('posts').select('*').range(0, 9) returns first 10 rows with Content-Range
• Schema selection via Accept-Profile header works
• .single() returns one object (not array) and 406 on 0 or 2+ results
• .maybeSingle() returns one object or null

3. CI Gate

• All unit tests run in cargo test --workspace
• Parser tests are pure (no DB needed) and run on every PR
• Handler integration tests that require Postgres are documented and gated appropriately
• cargo clippy --workspace -- -D warnings passes with no new warnings