cloudlysis/observability/INSTRUMENTATION.md

Instrumentation Requirements (Control Plane Dashboards)

Global Conventions

• Build/version labeling: every service exports a gauge metric *_build_info{service,version,git_sha} 1
• Correlation identifiers:
  • Logs include correlation_id and request_id fields for all request spans
  • Traces propagate traceparent end-to-end and expose trace IDs in logs
• Cardinality safety:
  • tenant_id must not be a label on high-frequency metrics unless bounded (sampling, rollups, or explicit allowlist)

Dashboard: Noisy Neighbor & Tenant Health

Required metrics (examples):

• http_request_duration_ms_bucket{service,route,method,status} histogram
• job_duration_ms_bucket{job_kind,status} histogram for control-plane jobs
• Optional bounded tenant signals:
  • tenant_active_jobs{tenant_id} only if tenant count is bounded and enforced

Dashboard: API Regression & Deployment

Required metrics (examples):

• *_build_info{service,version,git_sha} 1
• http_request_duration_ms_bucket{service,route,method,status}
• http_requests_total{service,route,method,status} Deploy markers:
• Grafana annotations for deploy events (vertical markers) or a low-cardinality metric like:
  • deploy_event{service,version,git_sha} 1 (sparse, emitted once per deploy)

Dashboard: Storage & Event Bus Bottlenecks

Required metrics (examples):

• Storage:
  • process_resident_memory_bytes{service}
  • disk_io_time_seconds_total{device} (node-exporter)
  • mdbx_* or equivalent libmdbx metrics if exposed by storage services
• Event bus / JetStream:
  • nats_* / jetstream_* metrics for consumer lag, ack latency, stream bytes, and redeliveries

Dashboard: Infrastructure Exhaustion

Required metrics (examples):

• Node exporter:
  • node_cpu_seconds_total
  • node_memory_MemAvailable_bytes
  • node_filesystem_avail_bytes
• Container/service level:
  • process_open_fds
  • tokio_* runtime metrics (if enabled) for saturation indicators