awesome-copilot/skills/cloud-design-patterns/references/messaging-integration.md

# Messaging & Integration Patterns

## Choreography Pattern

**Problem**: Central orchestrators create coupling and single points of failure.

**Solution**: Let individual services decide when and how a business operation is processed through event-driven collaboration.

**When to Use**:
- Loosely coupled microservices architectures
- Event-driven systems
- Avoiding central orchestration bottlenecks

**Implementation Considerations**:
- Use publish-subscribe messaging for event distribution
- Each service publishes domain events and subscribes to relevant events
- Implement saga pattern for complex workflows
- Ensure idempotency as events may be delivered multiple times
- Provide comprehensive logging and distributed tracing

## Claim Check Pattern

**Problem**: Large messages can overwhelm message infrastructure.

**Solution**: Split a large message into a claim check (reference) and a payload stored separately.

**When to Use**:
- Messages exceed messaging system size limits
- Reducing message bus load
- Handling large file transfers asynchronously

**Implementation Considerations**:
- Store payload in blob storage or database
- Send only reference/URI through message bus
- Implement expiration policies for stored payloads
- Handle access control for payload storage
- Consider costs of storage vs message transmission

## Competing Consumers Pattern

**Problem**: Single consumer may not keep up with message volume.

**Solution**: Enable multiple concurrent consumers to process messages from the same messaging channel.

**When to Use**:
- High message throughput requirements
- Scaling message processing horizontally
- Load balancing across multiple instances

**Implementation Considerations**:
- Ensure messages can be processed in any order
- Use competing consumer queues (Service Bus, RabbitMQ)
- Implement idempotency for message handlers
- Handle poison messages with retry and dead-letter policies
- Scale consumer count based on queue depth

## Messaging Bridge Pattern

**Problem**: Different systems use incompatible messaging technologies.

**Solution**: Build an intermediary to enable communication between messaging systems that are otherwise incompatible.

**When to Use**:
- Migrating between messaging systems
- Integrating with legacy systems
- Connecting cloud and on-premises messaging

**Implementation Considerations**:
- Transform message formats between systems
- Handle protocol differences
- Maintain message ordering if required
- Implement error handling and retry logic
- Monitor bridge performance and health

## Pipes and Filters Pattern

**Problem**: Complex processing tasks are difficult to maintain and reuse.

**Solution**: Break down a task that performs complex processing into a series of separate, reusable elements (filters) connected by channels (pipes).

**When to Use**:
- Processing data streams with multiple transformations
- Building reusable processing components
- Enabling parallel processing of independent operations

**Implementation Considerations**:
- Each filter performs a single transformation
- Connect filters using message queues or streams
- Enable parallel execution where possible
- Handle errors within filters or at pipeline level
- Support filter composition and reordering

## Publisher-Subscriber Pattern

**Problem**: Applications need to broadcast information to multiple interested consumers.

**Solution**: Enable an application to announce events to multiple consumers asynchronously, without coupling senders to receivers.

**When to Use**:
- Broadcasting events to multiple interested parties
- Decoupling event producers from consumers
- Implementing event-driven architectures

**Implementation Considerations**:
- Use topic-based or content-based subscriptions
- Ensure message delivery guarantees match requirements
- Implement subscription filters for selective consumption
- Handle consumer failures without affecting publishers
- Consider message ordering requirements per subscriber

## Scheduler Agent Supervisor Pattern

**Problem**: Distributed actions need coordination and monitoring.

**Solution**: Coordinate a set of actions across distributed services and resources with a supervisor that monitors and manages the workflow.

**When to Use**:
- Orchestrating multi-step workflows
- Coordinating distributed transactions
- Implementing resilient long-running processes

**Implementation Considerations**:
- Scheduler dispatches tasks to agents
- Agents perform work and report status
- Supervisor monitors progress and handles failures
- Implement compensation logic for failed steps
- Maintain state for workflow recovery