awesome-copilot/skills/cloud-design-patterns/references/reliability-resilience.md

Reliability & Resilience Patterns

Ambassador Pattern

Problem: Services need proxy functionality for network requests (logging, monitoring, routing, security).

Solution: Create helper services that send network requests on behalf of a consumer service or application.

When to Use:

• Offloading common client connectivity tasks (monitoring, logging, routing)
• Supporting legacy applications that can't be easily modified
• Implementing retry logic, circuit breakers, or timeout handling for remote services

Implementation Considerations:

• Deploy ambassador as a sidecar process or container with the application
• Consider network latency introduced by the proxy layer
• Ensure ambassador doesn't become a single point of failure

Bulkhead Pattern

Problem: A failure in one component can cascade and affect the entire system.

Solution: Isolate elements of an application into pools so that if one fails, the others continue to function.

When to Use:

• Isolating critical resources from less critical ones
• Preventing resource exhaustion in one area from affecting others
• Partitioning consumers and resources to improve availability

Implementation Considerations:

• Separate connection pools for different backends
• Partition service instances across different groups
• Use resource limits (CPU, memory, threads) per partition
• Monitor bulkhead health and capacity

Circuit Breaker Pattern

Problem: Applications can waste resources attempting operations that are likely to fail.

Solution: Prevent an application from repeatedly trying to execute an operation that's likely to fail, allowing it to continue without waiting for the fault to be fixed.

When to Use:

• Protecting against cascading failures
• Failing fast when a remote service is unavailable
• Providing fallback behavior when services are down

Implementation Considerations:

• Define threshold for triggering circuit breaker (failures/time window)
• Implement three states: Closed, Open, Half-Open
• Set appropriate timeout values for operations
• Log state transitions and failures for diagnostics
• Provide meaningful error messages to clients

Compensating Transaction Pattern

Problem: Distributed transactions are difficult to implement and may not be supported.

Solution: Undo the work performed by a sequence of steps that collectively form an eventually consistent operation.

When to Use:

• Implementing eventual consistency in distributed systems
• Rolling back multi-step business processes that fail partway through
• Handling long-running transactions that can't use 2PC

Implementation Considerations:

• Define compensating logic for each step in transaction
• Store enough state to undo operations
• Handle idempotency for compensation operations
• Consider ordering dependencies between compensating actions

Retry Pattern

Problem: Transient failures are common in distributed systems.

Solution: Enable applications to handle anticipated temporary failures by retrying failed operations.

When to Use:

• Handling transient faults (network glitches, temporary unavailability)
• Operations expected to succeed after a brief delay
• Non-idempotent operations with careful consideration

Implementation Considerations:

• Implement exponential backoff between retries
• Set maximum retry count to avoid infinite loops
• Distinguish between transient and permanent failures
• Ensure operations are idempotent or track retry attempts
• Consider jitter to avoid thundering herd problem

Health Endpoint Monitoring Pattern

Problem: External tools need to verify system health and availability.

Solution: Implement functional checks in an application that external tools can access through exposed endpoints at regular intervals.

When to Use:

• Monitoring web applications and back-end services
• Implementing readiness and liveness probes
• Providing detailed health information to orchestrators

Implementation Considerations:

• Expose health endpoints (e.g., /health, /ready, /live)
• Check critical dependencies (databases, queues, external services)
• Return appropriate HTTP status codes (200, 503)
• Implement authentication/authorization for sensitive health data
• Provide different levels of detail based on security context

Leader Election Pattern

Problem: Distributed tasks need coordination through a single instance.

Solution: Coordinate actions in a distributed application by electing one instance as the leader that manages collaborating task instances.

When to Use:

• Coordinating distributed tasks
• Managing shared resources in a cluster
• Ensuring single-instance execution of critical tasks

Implementation Considerations:

• Use distributed locking mechanisms (Redis, etcd, ZooKeeper)
• Handle leader failures with automatic re-election
• Implement heartbeats to detect leader health
• Ensure followers can become leaders quickly

Saga Pattern

Problem: Maintaining data consistency across microservices without distributed transactions.

Solution: Manage data consistency across microservices in distributed transaction scenarios using a sequence of local transactions.

When to Use:

• Long-running business processes spanning multiple services
• Distributed transactions without 2PC support
• Eventual consistency requirements across microservices

Implementation Considerations:

• Choose between orchestration (centralized) or choreography (event-based)
• Define compensating transactions for rollback scenarios
• Handle partial failures and rollback logic
• Implement idempotency for all saga steps
• Provide clear audit trails and monitoring

Sequential Convoy Pattern

Problem: Process related messages in order without blocking independent message groups.

Solution: Process a set of related messages in a defined order without blocking other message groups.

When to Use:

• Message processing requires strict ordering within groups
• Independent message groups can be processed in parallel
• Implementing session-based message processing

Implementation Considerations:

• Use session IDs or partition keys to group related messages
• Process each group sequentially but process groups in parallel
• Handle message failures within a session appropriately