Skip to content

Debug (Log → Trace → Metrics)

Overview

This skill is for debugging with existing telemetry. It does not focus on adding instrumentation (use observability when telemetry gaps block triage).

Goal: turn “something is broken/slow” into:

  • a concrete symptom + impact statement,
  • an evidence-backed hypothesis (or a small set of competing ones),
  • a mitigation (rollback/flag/scale) when needed,
  • a short list of fix + follow-up tasks.

Inputs / Outputs

Inputs: Symptom description (what's failing/slow); access to logs, traces, and metrics; environment and time window. Outputs: Root cause diagnosis with evidence, mitigation action, fix plan, learning capture. Feeds follow-ups to observability, resilience, platform, or architecture.

Workflow

0) Establish ground truth (2–5 minutes)

Capture:

  • Environment (local/dev/staging/prod) and time window (start/end).
  • Symptom (what’s failing/slow) and impact (SLO/user-visible blast radius).
  • One exemplar: request/trace ID, job run ID, message ID, or timestamped log line.

1) Logs (find the exemplar and its correlation IDs)

  1. Find the first error/timeout log line closest to the symptom window.
  2. Identify correlation keys (prefer stable IDs):
  3. traceId, requestId, spanId
  4. op (route template / RPC method / job name / message type)
  5. error code/type (typed error envelope, gRPC status, HTTP status)
  6. Pull the full log story for the exemplar (start → downstream call(s) → failure).

Copy/paste helpers live in references/commands.md.

2) Trace (turn the exemplar into a dependency hypothesis)

If you have a traceId, use it.

  1. Open the trace and confirm the root span matches the suspected operation (op).
  2. Identify:
  3. the slowest span(s),
  4. the first error span(s),
  5. retries (multiple similar child spans),
  6. deadline/time budget signals (deadline exceeded, timeout errors).
  7. Convert that to a dependency statement:
  8. “service A is timing out calling service B method X”
  9. “DB query Y is slow / missing index / deadlocked”
  10. “Queue consumer is failing on message type T (poison message)”

If you cannot find/interpret traces, fall back to logs + metrics and consider adding missing telemetry via observability.

3) Metrics (confirm blast radius + regression)

Use metrics to answer:

  • Is this widespread or isolated to one tenant/route/method?
  • Is it a new regression (deploy-correlated) or a gradual degradation (resource/saturation)?
  • Is it primarily errors or latency?

Start with RED for the boundary (HTTP route / gRPC method / consumer group).

4) Map failure propagation (technical + organizational)

  • If this component degrades, what fails next?
  • What is likely failing silently (data drift, dropped work, partial writes, stale reads)?
  • What is the organizational cascade (handoff queue, approvals, unclear ownership)?

GATE: Failure propagation (step 4) must be mapped before deciding mitigate vs investigate. If you don't know what breaks next, you can't assess the urgency of mitigation.

5) Decide: mitigate vs investigate

If impact is high and evidence points to a recent change:

  • rollback / disable flag / reduce load / scale critical dependency

If impact is moderate or unclear:

  • tighten the hypothesis with 1–2 targeted checks (another exemplar trace, compare two instances, check downstream health)

6) Capture learnings (don’t lose the fix)

If you found a systemic gap, capture it:

  • missing telemetry field contracts → observability
  • retries without idempotency / missing time budgets → resilience
  • repeated boundary logic across services → platform
  • cross-service pattern confusion → architecture

Minimum viable execution

When context or time is constrained, these are the load-bearing steps:

  1. Establish ground truth (step 0) — environment, symptom, one exemplar.
  2. Follow log → trace → metrics (steps 1-3) — this sequence, in this order.
  3. Map failure propagation (step 4) — what breaks next, what breaks silently.
  4. Capture learning (step 6) — don't lose the fix; route to the right skill.

Steps that can be cut under pressure: detailed metric blast-radius analysis (step 3 depth), organizational cascade mapping (step 4 breadth).

Guardrails

  • Don’t log secrets/PII while triaging (even “temporarily”).
  • Don’t use unbounded IDs as metric labels; use logs/traces for per-entity investigation.
  • Don’t add retries as a debugging “fix” without idempotency/dedupe.
  • Prefer a small number of exemplars (2–3) over “grep everything forever”.

Common failure modes

  • Jumps straight to reading code instead of following log → trace → metrics — this skips the evidence and leads to hypothesis-driven debugging without data.
  • Skips blast radius assessment (step 3-4) — fixes the symptom without knowing what else is affected or failing silently.
  • Fixes the immediate symptom without capturing learning (step 6) — the same failure recurs because the systemic gap is never routed to the right skill.
  • Checks only one exemplar and generalizes — different exemplars often reveal different failure modes.

References

Output Template

When using this skill, return:

  • Symptom: what is failing/slow (include concrete ops: route/method/job/message type).
  • Impact: who/what is affected and how badly (errors %, latency p95, backlog size).
  • Time window: start/end and whether it correlates with deploy/config change.
  • Evidence: exemplar IDs + the key log/trace/metric observations.
  • Hypothesis: most likely cause + 1 alternative (if applicable).
  • Failure propagation: what breaks next, what breaks silently, organizational cascade points.
  • Mitigation: what you did / recommend doing now (rollback/flag/scale).
  • Fix plan: code/config changes to make it correct and durable.
  • Follow-ups: telemetry gaps, runbook updates, tests, new invariants.
  • If root cause is systemic, flag for a follow-up retrospective (../references/structured-thinking-templates.md — Retrospective / Postmortem).