# Rollback Plan: Billing Enforced Gating

This document outlines the emergency procedure for disabling subscription-based entitlement gating in the Cloudlysis Control Plane.

## Symptoms
- Tenants receiving `402 Payment Required` errors even with valid active subscriptions.
- `JobEngine` refusing valid configuration updates due to incorrect resource limit enforcement.
- S3 Document Storage being inaccessible for authorized Pro/Enterprise tenants.

## Emergency Rollback Steps

### 1. Disable Global Enforcement
The quickest way to restore service is to disable enforcement via the environment variable toggle. This preserves all billing data and synchronization logic but bypasses the "Payment Required" blocks.

```bash
# In your Swarm stack file (e.g. control-plane.yml):
services:
  control-api:
    environment:
      - CONTROL_BILLING_ENFORCEMENT_ENABLED=false
```

### 2. Deploy the Update
Deploy the stack to apply the change:
```bash
docker stack deploy -c control-plane.yml control
```

### 3. Verify System State
Confirm that tenants can now perform previously blocked operations (e.g., uploading documents or updating deployment configurations).

## Forensic Analysis
Once the system is stable, perform the following:
1.  **Check Reconciliation Logs**: Look for `failed to fetch subscription` or `failed to apply reconciled billing event`.
2.  **Verify Metrics**: Check `billing_webhook_requests_total{status="error"}` in Prometheus.
3.  **Audit Drift**: Compare the `CONTROL_BILLING_STATE_PATH` file content against the Stripe Dashboard for the affected `tenant_id`.

## Recovery
To re-enable gating (after the root cause is resolved):
1.  Set `CONTROL_BILLING_ENFORCEMENT_ENABLED=true`.
2.  Redeploy the stack.
3.  Monitor logs and metrics for 30 minutes.