awesome-copilot/skills/cloud-design-patterns/references/messaging-integration.md

Messaging & Integration Patterns

Choreography Pattern

Problem: Central orchestrators create coupling and single points of failure.

Solution: Let individual services decide when and how a business operation is processed through event-driven collaboration.

When to Use:

• Loosely coupled microservices architectures
• Event-driven systems
• Avoiding central orchestration bottlenecks

Implementation Considerations:

• Use publish-subscribe messaging for event distribution
• Each service publishes domain events and subscribes to relevant events
• Implement saga pattern for complex workflows
• Ensure idempotency as events may be delivered multiple times
• Provide comprehensive logging and distributed tracing

Claim Check Pattern

Problem: Large messages can overwhelm message infrastructure.

Solution: Split a large message into a claim check (reference) and a payload stored separately.

When to Use:

• Messages exceed messaging system size limits
• Reducing message bus load
• Handling large file transfers asynchronously

Implementation Considerations:

• Store payload in blob storage or database
• Send only reference/URI through message bus
• Implement expiration policies for stored payloads
• Handle access control for payload storage
• Consider costs of storage vs message transmission

Competing Consumers Pattern

Problem: Single consumer may not keep up with message volume.

Solution: Enable multiple concurrent consumers to process messages from the same messaging channel.

When to Use:

• High message throughput requirements
• Scaling message processing horizontally
• Load balancing across multiple instances

Implementation Considerations:

• Ensure messages can be processed in any order
• Use competing consumer queues (Service Bus, RabbitMQ)
• Implement idempotency for message handlers
• Handle poison messages with retry and dead-letter policies
• Scale consumer count based on queue depth

Messaging Bridge Pattern

Problem: Different systems use incompatible messaging technologies.

Solution: Build an intermediary to enable communication between messaging systems that are otherwise incompatible.

When to Use:

• Migrating between messaging systems
• Integrating with legacy systems
• Connecting cloud and on-premises messaging

Implementation Considerations:

• Transform message formats between systems
• Handle protocol differences
• Maintain message ordering if required
• Implement error handling and retry logic
• Monitor bridge performance and health

Pipes and Filters Pattern

Problem: Complex processing tasks are difficult to maintain and reuse.

Solution: Break down a task that performs complex processing into a series of separate, reusable elements (filters) connected by channels (pipes).

When to Use:

• Processing data streams with multiple transformations
• Building reusable processing components
• Enabling parallel processing of independent operations

Implementation Considerations:

• Each filter performs a single transformation
• Connect filters using message queues or streams
• Enable parallel execution where possible
• Handle errors within filters or at pipeline level
• Support filter composition and reordering

Publisher-Subscriber Pattern

Problem: Applications need to broadcast information to multiple interested consumers.

Solution: Enable an application to announce events to multiple consumers asynchronously, without coupling senders to receivers.

When to Use:

• Broadcasting events to multiple interested parties
• Decoupling event producers from consumers
• Implementing event-driven architectures

Implementation Considerations:

• Use topic-based or content-based subscriptions
• Ensure message delivery guarantees match requirements
• Implement subscription filters for selective consumption
• Handle consumer failures without affecting publishers
• Consider message ordering requirements per subscriber

Scheduler Agent Supervisor Pattern

Problem: Distributed actions need coordination and monitoring.

Solution: Coordinate a set of actions across distributed services and resources with a supervisor that monitors and manages the workflow.

When to Use:

• Orchestrating multi-step workflows
• Coordinating distributed transactions
• Implementing resilient long-running processes

Implementation Considerations:

• Scheduler dispatches tasks to agents
• Agents perform work and report status
• Supervisor monitors progress and handles failures
• Implement compensation logic for failed steps
• Maintain state for workflow recovery