Event Replay & DLQ Monitoring
Event-DrivenAdvancedReliability engineeringResilient event pipeline with parallel consumers, dead-letter parking, manual replay trigger, and PagerDuty alerting. Use to build observable, fault-tolerant event-driven systems.
Recommended for: Teams that need guaranteed event delivery
Scope Notes
Intended for Teams that need guaranteed event delivery at Advanced complexity and Reliability engineering scope. Treat the graph as a starting shape rather than a fixed stack: keep the topology and the signals below, then swap providers to match your platform.
Review Checklist
Use Case
Financial transaction processing, order fulfilment, notification systems with strict delivery guarantees
Best Fit Scenarios
- Pipelines where no event may be dropped
- Operations with manual replay requirements
- Systems with on-call alerting expectations
Stack Breakdown
Architecture Layers
Components by Category
backend
async
frontend
external
Why This Topology Works
Consumers process independently, and a failure parks the event in the DLQ instead of blocking the partition. PagerDuty is alerted immediately, and the replay worker lets an operator re-process failures at a controlled rate once the cause is fixed.
Scaling Notes
Kafka partitions scale consumers horizontally. DLQ retention is configured separately from the main topic, and replay concurrency is throttled so a backlog cannot overwhelm downstream services.
Observability
Monitor consumer lag, DLQ depth, replay success rate and time-to-recovery. DLQ growth beyond threshold is the paging signal.
Typical Bottlenecks
- Consumer lag during replay backlogs
- DLQ growth during upstream incidents
- Replay idempotency when failures are not deterministic
Async Flow and Reliability
Events stream through Kafka and both consumers run independently. Any failure routes to the DLQ, which alerts on-call and stays parked until an operator triggers a throttled replay.
Upgrade Path
Add operator authentication in front of the replay console and an idempotency guard keyed on a replay id, then add a retry topic ahead of the DLQ so first-time failures are not treated as terminal.