Skip to main content

Reliability Patterns: Introduction

Understanding why reliability patterns are essential and what separates production microservices from toy implementations.

Table of Contents​

  1. Why Reliability Patterns Matter
  2. The Fallacy of Perfect Systems
  3. Cascading Failures
  4. Building Resilient Systems
  5. Pattern Categories
  6. Reliability vs Availability
  7. Measuring Reliability

Why Reliability Patterns Matter​

The Toy vs Production Gap​

A toy microservice works when everything is perfect:

  • Network is fast and reliable
  • Services are always available
  • Resources are unlimited
  • Load is predictable
  • Dependencies never fail

A production microservice expects failure:

  • Networks have latency and packet loss
  • Services crash and restart
  • Resources are constrained
  • Load spikes unpredictably
  • Dependencies fail regularly

Reliability patterns bridge this gap.

Real-World Failures​

Production systems fail in countless ways:

Network Issues:

  • Packet loss
  • High latency
  • Connection timeouts
  • DNS resolution failures
  • Routing problems

Service Issues:

  • Process crashes
  • Memory leaks causing OOM
  • Event loop starvation
  • Thread pool exhaustion
  • Deadlocks

Infrastructure Issues:

  • Hardware failures
  • Power outages
  • Disk failures
  • Network switches failing
  • Load balancer issues

Operational Issues:

  • Deployment errors
  • Configuration mistakes
  • Resource exhaustion
  • Dependency outages
  • Human error

Without reliability patterns, any of these causes cascading failures affecting the entire system.

Cost of Downtime​

Downtime is expensive:

  • Revenue loss (e-commerce down = no sales)
  • Reputation damage (users lose trust)
  • Customer churn (users switch to competitors)
  • SLA violations (contractual penalties)
  • Engineering time (incident response and firefighting)

Example: For a service handling $1M/hour in transactions, 5 minutes of downtime costs $83,000 in lost revenue alone. Add reputation damage and engineering costs, and a single incident can cost hundreds of thousands.

Investing in reliability patterns prevents these costs.

Blast Radius​

Without reliability patterns, failures spread:

Scenario: Payment service is slow.

  • Order service times out waiting for payment
  • Order service requests pile up
  • Order service runs out of memory
  • Order service crashes
  • Load balancer routes to other order instances
  • Those instances also get overwhelmed
  • Entire order service becomes unavailable
  • Website can't take orders
  • Business is offline

With reliability patterns:

  • Payment service is slow
  • Order service detects slowness
  • Circuit breaker opens for payment service
  • Orders fail fast with clear error
  • Order service remains healthy
  • Other functionality (browsing, cart) works
  • Business degraded but operational

Reliability patterns contain failures instead of letting them cascade.

The Fallacy of Perfect Systems​

Distributed Systems Are Different​

In monoliths, function calls are reliable:

  • Nanosecond latency
  • Never fail (unless process crashes)
  • Return value or throw exception
  • Predictable performance

In distributed systems, network calls are unreliable:

  • Millisecond to second latency
  • Can fail in many ways
  • May not return at all
  • Unpredictable performance

You cannot treat network calls like function calls.

Partial Failures​

In distributed systems, parts of the system fail while others work. This never happens in monoliths.

Examples:

  • Database is down but cache works
  • Primary datacenter offline but secondary works
  • One microservice crashed but others running
  • Network between services is partitioned

Systems must handle partial failures gracefully.

The Eight Fallacies of Distributed Computing​

Engineers often assume:

  1. The network is reliable
  2. Latency is zero
  3. Bandwidth is infinite
  4. The network is secure
  5. Topology doesn't change
  6. There is one administrator
  7. Transport cost is zero
  8. The network is homogeneous

All are false. Reliability patterns address these realities.

Timeouts Are Not Optional​

Without timeouts:

  • Hung requests consume resources indefinitely
  • Thread pools and connection pools exhaust
  • New requests can't be processed
  • System becomes unresponsive
  • Recovery requires restart

Every external call needs a timeout. No exceptions.

Retries Can Make Things Worse​

Naive retries amplify load:

  • Service is slow due to load
  • Clients timeout and retry
  • Retry traffic increases load further
  • Service becomes slower
  • More timeouts and retries
  • Death spiral

Smart retries are essential. Naive retries are dangerous.

Cascading Failures​

How Cascades Happen​

Cascading failures follow a pattern:

Stage 1: Initial Failure - One component fails or degrades.

Stage 2: Resource Exhaustion - Callers wait for slow component, exhausting their resources.

Stage 3: Cascade - Callers fail, affecting their callers, spreading through the system.

Stage 4: Total Outage - Entire system becomes unavailable.

Example Cascade​

Minute 0: Database replica fails. Primary handles all traffic.

Minute 1: Primary is slower due to load. Query latency increases from 10ms to 100ms.

Minute 2: API services waiting for queries. Connection pools fill up.

Minute 5: API services out of connections. New requests wait.

Minute 7: API service memory fills with waiting requests. GC thrashing begins.

Minute 10: API services crash from OOM. Load balancer routes to remaining instances.

Minute 12: Remaining instances immediately overwhelmed. They crash too.

Minute 15: All API services down. Website offline.

Without reliability patterns, this cascade is inevitable.

Preventing Cascades​

Reliability patterns break the cascade:

Timeouts - Don't wait forever for slow dependencies. Fail fast.

Circuit Breakers - Stop calling failed dependencies. Give them time to recover.

Bulkheads - Isolate failures. Don't let one dependency exhaust all resources.

Load Shedding - Drop some requests under overload rather than failing all requests.

Graceful Degradation - Disable non-critical features to maintain core functionality.

Each pattern contains failures at a different level.

Building Resilient Systems​

Defense in Depth​

No single pattern provides complete resilience. Layer multiple patterns:

Connection Level:

  • Connection timeouts
  • Connection pooling with limits
  • Connection retry with backoff

Request Level:

  • Request timeouts
  • Idempotency
  • Circuit breakers

Service Level:

  • Bulkheads
  • Rate limiting
  • Health checks

System Level:

  • Load shedding
  • Graceful degradation
  • Multiple availability zones

Fail Fast Philosophy​

When failure is inevitable, fail fast:

  • Return error immediately
  • Don't waste resources on doomed requests
  • Free resources for requests that can succeed
  • Provide clear error messages

Slow failures are worse than fast failures. Users can retry fast failures. Slow failures exhaust resources and cascade.

Design for Failure​

Assume every dependency will fail:

  • What happens when database is down?
  • What happens when payment service is slow?
  • What happens when cache is unavailable?

For each dependency, design:

  • Timeout strategy
  • Retry strategy
  • Fallback behavior
  • Degraded functionality

Testing for Failure​

Don't wait for production to test reliability:

Chaos Engineering - Deliberately inject failures in production or staging.

Load Testing - Test behavior under extreme load.

Failure Injection - Simulate dependency failures during testing.

Game Days - Scheduled exercises where teams practice incident response.

Systems that aren't tested against failure will fail in production.

Pattern Categories​

Prevention Patterns​

Prevent failures from occurring:

  • Rate Limiting - Prevent overload by limiting request rate
  • Load Shedding - Drop excess load to maintain quality for remaining requests
  • Backpressure - Signal upstream to slow down

Detection Patterns​

Detect failures quickly:

  • Health Checks - Monitor service health
  • Timeouts - Detect hung operations
  • Circuit Breakers - Detect sustained failures

Recovery Patterns​

Recover from failures:

  • Retries - Retry transient failures
  • Exponential Backoff - Space out retries to avoid overwhelming services
  • Graceful Degradation - Maintain partial functionality

Containment Patterns​

Contain failures to limit blast radius:

  • Bulkheads - Isolate resources to prevent exhaustion
  • Circuit Breakers - Stop calling failed dependencies
  • Timeouts - Prevent hung requests from consuming resources

Patterns Work Together​

These patterns complement each other:

  • Timeouts detect failures quickly
  • Circuit breakers stop retry storms
  • Bulkheads prevent resource exhaustion
  • Retries with backoff handle transient failures
  • Load shedding prevents overload

Use multiple patterns for comprehensive resilience.

Reliability vs Availability​

Reliability​

Reliability is the probability a system will work correctly over time. Measured by Mean Time Between Failures (MTBF).

A highly reliable system rarely fails.

Availability​

Availability is the proportion of time a system is operational. Measured in "nines":

  • 99% (two nines) = 3.65 days downtime/year
  • 99.9% (three nines) = 8.76 hours downtime/year
  • 99.99% (four nines) = 52.56 minutes downtime/year
  • 99.999% (five nines) = 5.26 minutes downtime/year

Availability = MTBF / (MTBF + MTTR)

Where MTTR is Mean Time To Recovery.

The Relationship​

Increase reliability - Make failures less frequent (increase MTBF).

Increase availability - Recover faster from failures (decrease MTTR).

Reliability patterns help both:

  • Prevent failures → increase reliability
  • Detect and recover faster → increase availability

Designing for Availability​

High availability requires:

  • Redundancy (no single point of failure)
  • Health checks (detect failures quickly)
  • Automatic failover (switch to backup quickly)
  • Graceful degradation (maintain partial service)

Five nines availability is expensive and complex. Most systems don't need it. Choose target based on business impact of downtime.

Measuring Reliability​

SLIs, SLOs, and SLAs​

SLI (Service Level Indicator) - Quantitative measure of service level. Examples:

  • Request latency
  • Error rate
  • Throughput
  • Availability

SLO (Service Level Objective) - Target for SLI. Examples:

  • 99.9% of requests complete in under 500ms
  • Error rate below 0.1%
  • 99.95% availability

SLA (Service Level Agreement) - Contract with consequences if SLO not met. Customers receive compensation for SLA violations.

Error Budgets​

If SLO is 99.9% availability, you have 0.1% error budget (43 minutes/month).

Use error budget to balance velocity and reliability:

  • Budget remaining → move fast, take risks
  • Budget exhausted → slow down, focus on reliability

Monitoring Reliability​

Track:

  • Error rate (errors per second, percentage)
  • Latency (p50, p95, p99, p99.9)
  • Availability (uptime percentage)
  • Time to detect issues
  • Time to recovery

Alert on SLO violations, not arbitrary thresholds.

Reliability Culture​

Reliability is not just technical—it's cultural:

  • Engineers own reliability of their services
  • Incidents are learning opportunities, not blame
  • Reliability is part of feature development
  • Testing includes failure scenarios
  • Monitoring and alerting are first-class concerns

The Reliability Tax​

Reliability patterns add complexity:

  • More code to write and maintain
  • More configuration to manage
  • More testing scenarios
  • More monitoring and alerting

But the cost is far less than the cost of unreliability. Production-grade systems must pay this tax.

Starting Point​

For new systems:

  1. Start with timeouts on all external calls
  2. Add health checks and monitoring
  3. Implement circuit breakers for critical dependencies
  4. Add rate limiting at boundaries
  5. Implement retries with exponential backoff
  6. Add load shedding under overload
  7. Design graceful degradation paths

Don't try to implement everything at once. Build reliability incrementally, prioritizing highest-risk areas.