Skip to content

Scenario Checklists

Use these as a guide to keep triage systematic and fast.

HTTP: error spike (5xx) or latency regression

  1. Identify the operation:
  2. route template (not raw URL)
  3. status codes (4xx vs 5xx vs timeouts)
  4. Find 1–3 exemplar requests in logs:
  5. capture traceId / requestId, op, and top-level error envelope
  6. Use traces to locate the “first failure”:
  7. dependency call timing out?
  8. retry amplification?
  9. DB span slow/error?
  10. Use metrics to confirm scope:
  11. request rate change (traffic spike?)
  12. error rate and p95 latency by route (if available)
  13. saturation signals (CPU/mem/event loop/DB connections)
  14. Mitigation checklist:
  15. rollback latest deploy or disable flag
  16. shed load / rate limit if needed
  17. scale the dependency that’s saturated
  18. Fix checklist:
  19. add/adjust timeouts and propagate cancellation
  20. eliminate unsafe retries (or add idempotency/dedupe)
  21. optimize DB query or add index (validate via span timing)

gRPC: DEADLINE_EXCEEDED / slow method

  1. Identify:
  2. grpc_service + grpc_method
  3. caller and server services
  4. Logs:
  5. look for DEADLINE_EXCEEDED, UNAVAILABLE, connection resets
  6. confirm call deadlines/time budgets are set and propagated
  7. Traces:
  8. confirm client/server spans are linked (if using trace context propagation)
  9. find the slowest server span and the slowest downstream span
  10. Metrics:
  11. error rate by grpc_code
  12. latency p95 by method
  13. Common causes:
  14. missing deadline propagation
  15. server threadpool/concurrency exhaustion
  16. downstream dependency latency (DB/cache/HTTP)
  17. retry amplification without idempotency

Async consumer: backlog / lag / poison messages

  1. Identify:
  2. consumer group / worker name
  3. message type / topic / stream
  4. Metrics:
  5. backlog depth / lag (if exported)
  6. processing rate vs incoming rate
  7. DLQ rate (if applicable)
  8. Logs:
  9. repeated failures on the same message key/ID?
  10. deserialization/validation errors (input contract drift)?
  11. timeouts calling downstream dependencies?
  12. Traces (if you propagate context):
  13. confirm each message run has a trace root span
  14. look for repeated retries and long-running spans
  15. Mitigation checklist:
  16. pause ingestion (if possible) or slow producers
  17. quarantine poison messages (DLQ) to restore throughput
  18. scale consumers (only if you’re not hitting a downstream limit)
  19. Fix checklist:
  20. validate inputs at the consumer boundary (treat as unknown)
  21. ensure idempotency/dedupe (at-least-once delivery is normal)
  22. add per-message timeouts + cancellation