Skip to content

Resilience

Overview

Make I/O failures boring: set explicit timeouts, retry safely, keep operations idempotent, and prevent cascades with circuit breakers and bulkheads.

Enterprise systems fail partially (timeouts, 5xx, queue lag). Resilience patterns make those failures bounded and observable.

Inputs / Outputs

Inputs: I/O boundary code (HTTP/gRPC/DB/cache/queue clients); dependency map (what calls what). Outputs: Hardened code with resilience patterns applied (timeouts, retries, idempotency, breakers, bulkheads); failure model documentation. Consumed by testing and finish.

Workflow

  1. Identify the I/O boundary: HTTP, gRPC, DB, cache, queue/stream, third-party API.
  2. Define the failure model:
  3. which failures are expected/transient vs permanent
  4. what “success” means under degradation (fallback? partial data? fail fast?)
  5. Apply patterns in this order:
  6. Timeouts + cancellation

  7. Idempotency (especially if retries exist)

    GATE: If adding retries, idempotency (pattern 2) MUST be addressed first. Never retry a non-idempotent operation without an idempotency key or server-side dedupe.

  8. Retries with backoff + jitter (bounded)

  9. Circuit breaker (when a dependency is unhealthy)
  10. Bulkheads / concurrency limits (to protect your own resources)
  11. Add observability (retry counts, breaker state, queue lag, error codes). Each metric should have a named decision it supports and an owner — see observability.
  12. Add consumer-visible tests for semantics; add a local smoke test for failure modes.

Minimum viable execution

When context or time is constrained, these are the load-bearing steps:

  1. Identify I/O boundaries (step 1) — what calls what.
  2. Apply timeout + cancellation (step 3.1) — mandatory for every outbound call.
  3. Ensure idempotency before retries (step 3.2-3.3) — never retry without it.
  4. Verify (step 5) — test that timeout errors are stable and retries are bounded.

Steps that can be cut under pressure: circuit breaker (step 3.4), bulkheads (step 3.5), detailed observability (step 4).

Chooser (What To Use Where)

  • Inbound HTTP/gRPC: timeouts for downstream calls; guardrails; load shedding; return stable error codes.
  • Outbound HTTP/gRPC clients: timeouts + bounded retries + (optional) circuit breaker + concurrency limit.
  • DB/Redis: short timeouts; limited retries (often none); concurrency limits/pools; backpressure.
  • Message consumers: idempotency + dedupe; retry with backoff; dead-letter strategy; track lag.

Clarifying Questions

  • What I/O boundary is involved (HTTP client, gRPC client, DB, cache, queue/stream, third-party API)?
  • What failures are expected/transient vs permanent for this dependency?
  • Is the operation idempotent? If not, can you add an idempotency key?
  • What is the time budget for this operation (how long can the caller wait)?
  • What should happen under degradation (fail fast, return stale/cached data, partial results, fallback)?
  • Are there existing resilience patterns in the codebase (retry helpers, circuit breaker, timeout wrappers)?

Core Patterns (Opinionated Defaults)

Timeouts + cancellation (mandatory)

  • Every outbound call has a timeout and participates in cancellation.
  • Use a single “time budget” per request; don’t let retries exceed it.

Retries (bounded, only when safe)

  • Retry only for transient failures (timeouts, connection resets, 429/503 depending on semantics).
  • Use exponential backoff + jitter to avoid synchronized retry storms.
  • Cap attempts and cap total delay; consider a retry budget per request.

Idempotency (required when retries exist)

  • Never retry a non-idempotent operation unless the server-side operation is idempotent (idempotency key / dedupe).
  • For message consumers, assume at-least-once delivery: handle duplicates safely.

Circuit breaker (for flakey/unhealthy dependencies)

  • Open the breaker when failure rate exceeds a threshold; fail fast for a cooldown period.
  • Half-open to probe recovery; close after successful probes.
  • Expose breaker state as metrics and log transitions.

Bulkheads / concurrency limits (protect yourself)

  • Limit concurrent work per dependency to avoid saturating your process.
  • Prefer per-dependency limits (bulkheads) over one global limit.

Minimal TypeScript Snippets

Backoff with jitter:

export function backoffDelayMs(
  attempt: number,
  baseMs = 100,
  maxMs = 2_000,
): number {
  const exp = Math.min(maxMs, baseMs * 2 ** attempt);
  const jitter = 0.5 + Math.random(); // [0.5, 1.5)
  return Math.round(exp * jitter);
}

Testing / Verification

  • Timeouts: request returns a stable timeout error within the budget.
  • Retries: attempts are bounded; retryable vs non-retryable errors are classified correctly.
  • Idempotency: duplicate request/message does not double-apply side effects.
  • Bulkheads: dependency overload doesn’t starve unrelated work.
  • Breaker: opens under repeated failures and closes after recovery.

Common failure modes

  • Adds retries without idempotency — creates duplicate side effects (double charges, duplicate messages, repeated writes).
  • Sets timeouts too high (e.g., 30s for an API call in a 5s request budget) — effectively no timeout; the caller hangs.
  • Applies circuit breaker to every dependency — circuit breakers add complexity and should only be used for genuinely flakey/unhealthy dependencies, not stable ones.
  • Ignores cancellation propagation — the caller times out but the downstream work continues, wasting resources.

References

Output Template

When applying this skill, return:

  • The failure model assumptions (retryable errors, idempotency strategy, time budget).
  • The minimal code changes (timeouts, retry loop, breaker/bulkhead wrapper).
  • The verification steps (tests + how to simulate failure locally).