Day 11 of #DistributedDilemmas: Queue-based architectures hide scaling issues

Kafka topics with long retention become both a blessing and a curse. Backpressure beats unbounded queues!

Kafka’s Retention Double-Edged Sword:
While queues create an illusion of infinite scalability by buffering load and smoothen temporary spikes, unbounded queues lead to hidden technical debt:

• Blessing: Long retention enables event replay for recovery/analytics (critical for financial transaction auditing).
• Curse: Segmentation Trap, Kafka only deletes closed log segments where active segments with expired data remain until rotation.

SlackHQ engineering team found 2-week retention caused 45TB storage bloat, slowing consumer bootstrap from 2s to 1m+. https://slack.engineering/scaling-slacks-job-queue/

---

Backpressure Patterns Beat Unbounded Queues:

• Maintain stable throughput under load (94% fewer OOM crashes at voxer : https://engineering.voxer.com/2013/09/16/backpressure-in-nodejs/)
• Enable native flow control without client-side retry storms (https://www.tedinski.com/2019/03/05/backpressure.html)
• Prevent consumer lag death spiral.

Day 11 on X ↗