161 lines
14 KiB
Markdown
161 lines
14 KiB
Markdown
### 🧱 Component: Aggregate
|
|
**Definition:**
|
|
The Aggregate is a standalone Rust-based container that serves as the primary consistency boundary and decision-making unit of the system. It is a stateful entity that encapsulates business logic, enforces invariants, and ensures that all changes to the system are valid according to defined rules. Commands are received from users through a Gateway, and events are stored on **NATS JetStream**; `edge-storage` `AggregateStore` holds versioned **snapshots** for efficient rehydration.
|
|
|
|
**Multi-Tenancy:**
|
|
The Aggregate supports optional multi-tenancy via `tenant_id`. When enabled:
|
|
- **Routing:** The Gateway routes commands to Aggregate nodes based on the `x-tenant-id` header
|
|
- **Sharding:** Aggregate instances are sharded across nodes by `tenant_id`, ensuring tenant data isolation
|
|
- **Storage:** Snapshots and events are namespaced by `tenant_id` to prevent cross-tenant access
|
|
- **Subject Naming:** NATS subjects include `tenant_id` (e.g., `tenant.<tenant_id>.aggregate.<aggregate_type>.<aggregate_id>`)
|
|
- **Backward Compatibility:** Aggregates without multi-tenancy use a default/empty `tenant_id`
|
|
|
|
**Dependencies:**
|
|
* Core crates pulled from the custom Cargo registry:
|
|
```toml
|
|
[registries.madapes]
|
|
index = "sparse+https://git.madapes.com/api/packages/madapes/cargo/"
|
|
```
|
|
|
|
| Crate | Purpose |
|
|
|-------|---------|
|
|
| `edge-storage` | libmdbx-backed AggregateStore for versioned snapshots |
|
|
| `runtime-function` | Deterministic DAG execution for `decide`/`apply` programs |
|
|
| `edge-logger` | High-performance logging (UDS + Protobuf, Loki sink) |
|
|
| `query-engine` | UQF query support for filtering/querying aggregate state |
|
|
| `async-nats` | NATS JetStream client for event streaming |
|
|
|
|
* Source code available at `../../madapes/`
|
|
* **Note:** This is a standalone container — it does not use `framework-bus` or `framework-aggregate` (those serve a different system)
|
|
|
|
**Observability:**
|
|
* Production stack: **Grafana** + **Victoria Metrics** + **Loki**
|
|
* `edge-logger` provides structured logging via Unix Domain Sockets with lock-free batching
|
|
* Metrics exposed via `metrics-exporter-prometheus` for Victoria Metrics scraping
|
|
* Traces/logs flow to Loki with cardinality protection and multi-tenant isolation
|
|
|
|
#### 1. Core Responsibilities
|
|
* **Command Validation:** Receives intent (Commands) from the Gateway and uses `runtime-function` DAG programs to determine if the intent is valid based on the current state.
|
|
* **State Rehydration:** Reconstructs its internal state by loading the latest **snapshot** from `edge-storage` `AggregateStore` (`get_latest_snapshot`) and replaying any subsequent events from NATS JetStream.
|
|
* **Event Production:** Transforms valid commands into one or more Events that represent a "fact" that has occurred.
|
|
* **Atomic Persistence:** Publishes new events to NATS JetStream and stores an updated snapshot in `edge-storage` `AggregateStore` (`put_snapshot_sync`).
|
|
* **Concurrency Control:** Protects against "lost updates" using version-based optimistic locking. `edge-storage` `AggregateStore` returns `VersionConflict` for duplicate versions.
|
|
|
|
#### 2. The Lifecycle of a Command
|
|
1. **Reception:** The Gateway routes a Command from a user to the Aggregate container based on the `aggregate_id` and `x-tenant-id` header. The `tenant_id` is extracted and included in the Command envelope for tenant-aware processing.
|
|
2. **Loading (Rehydration):**
|
|
* The Aggregate fetches the latest **Snapshot** from `edge-storage` `AggregateStore` using the composite key `(tenant_id, aggregate_id)`.
|
|
* It reads any **Events** from NATS JetStream (tenant-namespaced subject) that occurred after the snapshot version.
|
|
* It applies these events sequentially to the snapshot state using the deterministic `apply` runtime-function program to reach the "Current State."
|
|
3. **Execution:**
|
|
* The Aggregate passes the Current State and the Command to the `decide` runtime-function program.
|
|
* If invalid: Returns an Error (Command Rejected).
|
|
* If valid: Returns a list of New Events.
|
|
4. **Persistence (The Commit):**
|
|
* The Aggregate publishes New Events to NATS JetStream on tenant-namespaced subjects, with `command_id` mapped to `idempotency_key`.
|
|
* It stores an updated snapshot in `edge-storage` `AggregateStore` using `(tenant_id, aggregate_id, new_version)` as the composite key.
|
|
* **Constraint:** `AggregateStore` enforces strict monotonicity — if `new_version` already exists, it returns `VersionConflict`, and the Aggregate must reload and retry.
|
|
5. **Publication:**
|
|
* Events published to NATS JetStream are immediately available for downstream consumption by Sagas and Projections (filtered by tenant if needed).
|
|
|
|
#### 3. Technical Constraints & Guarantees
|
|
* **Determinism:** The logic within an Aggregate must be 100% deterministic. `runtime-function` DAG programs are sandboxed and gas-metered, with no access to the system clock, random number generators, or external APIs. All data required for a decision must be present in the Command or the Aggregate State.
|
|
* **Side-Effect Free:** An Aggregate does not send emails, update databases, or call other services. It only produces events. Side effects are the responsibility of Sagas.
|
|
* **Single Writer:** While multiple nodes may attempt to process commands for the same `aggregate_id`, only one "Commit" can succeed for a specific version, enforced by `edge-storage` `AggregateStore` (`VersionConflict`).
|
|
* **Tenant Isolation:** An Aggregate can only access data within its `tenant_id` scope. Cross-tenant access is blocked at the storage and stream layers. The `tenant_id` is validated on every command to prevent tenant spoofing.
|
|
* **Isolation:** An Aggregate cannot see the state of other Aggregates. If a business rule spans multiple Aggregates, it must be handled by a **Saga**.
|
|
|
|
#### 4. Data Structure (The Envelope)
|
|
Each Aggregate maintains a metadata header:
|
|
* `tenant_id`: Optional identifier for multi-tenant isolation (routed via `x-tenant-id` header)
|
|
* `aggregate_id`: Unique UUID or URN for the instance.
|
|
* `aggregate_type`: The name of the business entity (e.g., `Account`, `Order`).
|
|
* `version`: A monotonically increasing integer representing the number of events processed.
|
|
* `snapshot_threshold`: A configuration defining how many events should trigger a new snapshot in `edge-storage`.
|
|
|
|
#### 5. Error Handling
|
|
* **Validation Errors:** Business rule violations (e.g., "Insufficient Funds") result in an immediate synchronous rejection of the command.
|
|
* **Tenant Access Errors:** Cross-tenant access attempts (e.g., wrong `tenant_id` in command) are rejected with `TenantAccessDenied`.
|
|
* **Concurrency Conflicts:** If `edge-storage` returns `VersionConflict`, the framework implements an automatic "Retry-on-Conflict" policy (Reload → Re-validate → Re-commit) up to a defined limit.
|
|
* **System Failures:** If `edge-storage` or NATS JetStream is unavailable, the Aggregate remains in a read-only or "unavailable" state to prevent inconsistent branching of the event stream.
|
|
|
|
#### 6. Horizontal Scaling Strategy
|
|
The Aggregate container is designed for horizontal scaling on **Docker Swarm**, leveraging tenant-based sharding for predictable data locality and simple operations.
|
|
|
|
**Sharding Model:**
|
|
- **Tenant-Aware Placement:** Aggregate instances are placed on Swarm nodes based on `tenant_id` using Docker Swarm placement constraints
|
|
- **Consistent Hashing:** A hash ring maps `tenant_id` values to specific nodes, ensuring all commands for a tenant route to the same node (or replica set)
|
|
- **Subject-Based Routing:** NATS JetStream consumer groups are tenant-namespaced, enabling parallel processing across tenants without coordination
|
|
|
|
**Scaling Architecture:**
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Admin UI (Control Node) │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ Scale Manager: CRUD for tenant → node assignments │ │
|
|
│ │ - List tenants, node assignments, load metrics │ │
|
|
│ │ - Add/remove nodes, migrate tenants │ │
|
|
│ │ - Emit scaling commands to Docker Swarm API │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
└──────────────────────────┬──────────────────────────────────────┘
|
|
│ Docker Swarm API / SSH
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Docker Swarm Cluster │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Node A │ │ Node B │ │ Node C │ │
|
|
│ │ tenant: a-c │ │ tenant: d-m │ │ tenant: n-z │ │
|
|
│ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │
|
|
│ │ │Agg Ctr │ │ │ │Agg Ctr │ │ │ │Agg Ctr │ │ │
|
|
│ │ └───┬────┘ │ │ └───┬────┘ │ │ └───┬────┘ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │
|
|
│ │ ┌───▼────┐ │ │ ┌───▼────┐ │ │ ┌───▼────┐ │ │
|
|
│ │ │libmdbx │ │ │ │libmdbx │ │ │ │libmdbx │ │ │
|
|
│ │ │(local) │ │ │ │(local) │ │ │ │(local) │ │ │
|
|
│ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
│ │ │ │ │
|
|
│ └──────────────────┴──────────────────┘ │
|
|
│ │ │
|
|
│ ┌────────────────────────▼────────────────────────────────────┐ │
|
|
│ │ Shared NATS JetStream Cluster │ │
|
|
│ │ (tenant-namespaced subjects for isolation) │ │
|
|
│ └─────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
**Note:** Each node has its own embedded `edge-storage` (libmdbx) containing snapshots for its assigned tenants. NATS JetStream provides shared event storage. Tenant migration requires snapshot data transfer between nodes.
|
|
|
|
**Operational Model:**
|
|
- **Scale Up:** Admin UI calls Swarm API to add new node, updates tenant → node mapping, Gateway updates routing table
|
|
- **Scale Down:** Migrate tenants to other nodes (drain), remove node from Swarm
|
|
- **Tenant Migration:** Pause consumer, copy tenant data, update routing, resume on new node
|
|
- **Zero-Downtime:** New tenant assignments are picked up by Gateway via config reload without restart
|
|
|
|
**Placement Constraints:**
|
|
- Each Aggregate service runs with `--constraint node.labels.tenant_range==<range>`
|
|
- Gateway uses tenant → node mapping to route commands to correct Swarm service endpoint
|
|
- Multiple replicas per tenant range supported for HA (active-passive via NATS consumer groups)
|
|
|
|
**Admin Endpoints (per Aggregate container):**
|
|
- `/health` - Container health (NATS, storage, active aggregates)
|
|
- `/ready` - Readiness for receiving commands
|
|
- `/metrics` - Prometheus metrics with tenant_id labels
|
|
- `/admin/tenants` - List tenants hosted on this node (read-only)
|
|
- `/admin/drain` - Graceful drain for tenant migration
|
|
- `/admin/reload` - Hot-reload tenant placement config
|
|
|
|
**External Control Node:**
|
|
- Separate service that calls Aggregate admin endpoints
|
|
- Manages Docker Swarm API for scaling operations
|
|
- Publishes tenant → node mapping to NATS KV
|
|
- See Admin UI repository for full implementation
|
|
|
|
---
|
|
|
|
### 💡 Implementation Note:
|
|
The **Aggregate Logic** is a pair of `runtime-function` DAG programs:
|
|
1. **`decide` program**: `(state, command) → events[]` — The business logic (validates command, produces events).
|
|
2. **`apply` program**: `(state, event) → new_state` — The state transition logic (used during rehydration from snapshots + events).
|
|
|
|
These are referenced in the manifest as `decide:` and `apply:` fields under each aggregate definition.
|