Rollback Plan: Billing Enforced Gating

This document outlines the emergency procedure for disabling subscription-based entitlement gating in the Cloudlysis Control Plane.

Symptoms

Tenants receiving 402 Payment Required errors even with valid active subscriptions.
JobEngine refusing valid configuration updates due to incorrect resource limit enforcement.
S3 Document Storage being inaccessible for authorized Pro/Enterprise tenants.

Emergency Rollback Steps

1. Disable Global Enforcement

The quickest way to restore service is to disable enforcement via the environment variable toggle. This preserves all billing data and synchronization logic but bypasses the "Payment Required" blocks.

# In your Swarm stack file (e.g. control-plane.yml):
services:
  control-api:
    environment:
      - CONTROL_BILLING_ENFORCEMENT_ENABLED=false

2. Deploy the Update

Deploy the stack to apply the change:

docker stack deploy -c control-plane.yml control

3. Verify System State

Confirm that tenants can now perform previously blocked operations (e.g., uploading documents or updating deployment configurations).

Forensic Analysis

Once the system is stable, perform the following:

Check Reconciliation Logs: Look for failed to fetch subscription or failed to apply reconciled billing event.
Verify Metrics: Check billing_webhook_requests_total{status="error"} in Prometheus.
Audit Drift: Compare the CONTROL_BILLING_STATE_PATH file content against the Stripe Dashboard for the affected tenant_id.

Recovery

To re-enable gating (after the root cause is resolved):

Set CONTROL_BILLING_ENFORCEMENT_ENABLED=true.
Redeploy the stack.
Monitor logs and metrics for 30 minutes.

1.7 KiB Raw Blame History