Triage Commands (Copy/Paste)¶

These are intentionally generic templates. Replace placeholders like <service>, <namespace>, <route>, <method>, <traceId>, <requestId>.

Logs¶

Local files (JSON logs)¶

rg -n '"level":"error"|\"err\"|timeout|ECONNRESET|ETIMEDOUT|deadline exceeded' -S .

Filter JSON logs by traceId or requestId:

cat <logfile> | jq -c 'select(.traceId=="<traceId>" or .requestId=="<requestId>")'

Docker Compose¶

docker compose logs --since=10m <service>

Search within logs:

docker compose logs --since=10m <service> | rg -n 'error|timeout|traceId|requestId' -S

Kubernetes¶

kubectl -n <namespace> get pods -l app=<service>
kubectl -n <namespace> logs deploy/<service> --since=10m

Search for a trace/request ID:

kubectl -n <namespace> logs deploy/<service> --since=30m | rg -n '<traceId>|<requestId>' -S

Split by container (if multi-container pod):

kubectl -n <namespace> logs deploy/<service> -c <container> --since=10m

Traces¶

Trace UI (Tempo / Jaeger / vendor APM)¶

If you have a traceId, search for it in your trace UI and inspect:

slowest span(s)
first error span(s)
retries (repeated similar spans)
deadline/timeout signals (DEADLINE_EXCEEDED, ETIMEDOUT)

Tempo HTTP API (optional)¶

If you have Tempo reachable locally:

curl -sS "http://<tempo-host>:3200/api/traces/<traceId>" | jq '.'

List services involved (heuristic; schema varies by Tempo version):

curl -sS "http://<tempo-host>:3200/api/traces/<traceId>" \
  | jq -r '.. | objects | .resource?.attributes? // empty | .[]? | select(.key=="service.name") | .value?.stringValue' \
  | sort -u

Metrics (Prometheus-compatible)¶

Quick “is it on fire?” check¶

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=up{job="<service>"}'

HTTP RED (examples)¶

Request rate:

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=sum(rate(http_server_requests_total{service="<service>"}[5m]))'

Error rate:

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=sum(rate(http_server_requests_total{service="<service>",status=~"5.."}[5m]))'

p95 latency (histogram):

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{service="<service>"}[5m])))'

gRPC RED (examples)¶

RPC rate and errors:

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=sum(rate(grpc_server_handled_total{grpc_service="<Service>",grpc_method="<Method>"}[5m]))'

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=sum(rate(grpc_server_handled_total{grpc_service="<Service>",grpc_method="<Method>",grpc_code!="OK"}[5m]))'

Saturation / resource hints (examples)¶

Pod CPU/memory (k8s metrics-server required):

kubectl -n <namespace> top pods -l app=<service>

Node.js event loop lag (if exported):

curl -sS -G "http://<prom-host>:9090/api/v1/query" \
  --data-urlencode 'query=max(nodejs_eventloop_lag_p99_seconds{service="<service>"})'

Repro / boundary checks¶

HTTP¶

curl -v --max-time 10 "http://<host>:<port><route>"

gRPC (grpcurl)¶

List services:

grpcurl -plaintext <host>:<port> list

Call a method:

grpcurl -plaintext -d '{}' <host>:<port> <package.Service>/<Method>

Async consumers (common hints)¶

These depend on your queue/broker. Examples:

Redis Streams:
redis-cli XLEN <stream>
redis-cli XINFO GROUPS <stream>
Kafka:
kafka-consumer-groups --bootstrap-server <broker> --describe --group <group>