madbase/.trae/documents/plan_20260311_224831.md

Implement Missing Phase 2 Features

I will implement the remaining features for Phase 2: Advanced Replication (Realtime) and Resumable Uploads (Storage).

1. Advanced Realtime Replication (pgoutput)

Goal: Replace the LISTEN/NOTIFY fallback with robust logical replication using the pgoutput protocol.

• Dependencies: Add pgoutput crate and enable replication feature for tokio-postgres.
• Implementation:
  • Update realtime/src/replication.rs to connect to Postgres in replication mode.
  • Create a replication slot (madbase_slot) and start streaming from publication (madbase_pub).
  • Use pgoutput::Decoder to parse binary replication messages (Relation, Insert, Update, Delete).
  • Maintain an in-memory cache of Relation metadata (schema, table, columns) to map relation IDs to names.
  • Construct PostgresPayload from change events and broadcast to WebSocket clients.

2. Resumable Uploads (TUS Protocol)

Goal: Implement the TUS protocol for reliable large file uploads in the Storage service.

• Dependencies: Add base64 to storage/Cargo.toml.
• New Module: Create storage/src/tus.rs.
• Endpoints:
  • POST /storage/v1/upload/resumable: Initialize upload. Creates a local tracking file.
  • PATCH /storage/v1/upload/resumable/:id: Append data chunk to the local file.
  • HEAD /storage/v1/upload/resumable/:id: Return current upload offset.
• Completion Logic:
  • When offset == size, stream the complete file to S3.
  • Insert metadata into storage.objects.
  • Clean up local temporary files.

Execution Steps

1. Update Dependencies: Modify realtime/Cargo.toml and storage/Cargo.toml.
2. Implement Realtime Replication: Rewrite realtime/src/replication.rs with pgoutput logic.
3. Implement TUS Handlers: Create storage/src/tus.rs and register routes in storage/src/lib.rs.
4. Verify: Ensure compilation and check for basic logic correctness.