[Перевод] Компактация лога в Apache Kafka повреждала данные. Вот как мы это исправили
Компактация лога в Kafka может привести к ситуации, когда разные реплики одного топика начинают хранить разную версию данных — и расхождение сохраня…
Open Source
⚑ Report a ProblemLatest Open Source news from Tech News
All topics
AI
agents
ai
api
architecture
automation
backend
beginners
career
database
devchallenge
devops
discuss
javascript
linux
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Designing High-Throughput Event-Driven Microservices with Kafka and RabbitMQ
Introduction & Industry Context In the landscape of modern software architecture, event-driven microservices have transitioned from an advanced…
Microservices Event Pipelines: Outbox Pattern, CDC, and Idempotent Event Sourcing
In distributed microservice architectures, synchronizing database updates with downstream event streams is a notorious source of silent data drift.…
Kafka Partitions and Consumer Groups: How to Prevent Head-of-Line Blocking
Apache Kafka is celebrated for its incredible horizontal throughput, capable of streaming millions of messages per second. Yet, many engineering tea…
ClickHouse под трейсы: почему шесть бэкендов Jaeger оказались выбором из двух
Cloud Tracing — сервис распределённой трассировки в VK Cloud . Приложения отправляют в него спаны по протоколу OpenTelemetry, сервис хранит их и отд…
ClickHouse под трейсы: почему шесть бэкендов Jaeger оказались выбором из двух
Cloud Tracing — сервис распределённой трассировки в VK Cloud . Приложения отправляют в него спаны по протоколу OpenTelemetry, сервис хранит их и отд…
Integration Digest for September 2026
Articles 🔍 API Features as a First-Class Artifact with OpenAPI Overlays OpenAPI Overlay 1.1 documents model API features as separate, reviewable art…
Read Kafka Like a Database
Working with Kafka data on your own terms, using a local index. A search stack usually follows Kafka into a service. But the message you are looking…
2,404,811 answers on port 2181 and 1,381,208 on port 9092: the difference between a coordinator and a broker
2,404,811 answers on port 2181 and 1,381,208 on port 9092: the difference between a coordinator and a broker The counts Two ports associated with di…
Sidekiq to Kafka: A Mental-Model Map for Rails Developers
TL;DR: If you've spent years with Sidekiq, Kafka can look like "Sidekiq with extra steps". It isn't. Sidekiq moves jobs (commands to do something).…
One message we could not parse stopped four hours of dispatches
At twelve minutes past four on a Saturday morning, one partition of our shipments topic stopped moving. We found out at nine, when a depot rang to a…
От сырых проверок качества данных до понятной картины в OpenMetadata
Ни для кого не секрет, что проверки качества — фундамент доверия к данным. Пока вы не знаете, насколько полны и корректны ваши данные, любые основан…
Queues, Webhooks and Rate Limits: Retries, Backlogs and Backoff You Can Watch Happen
"At-least-once delivery" is four words that hide a dozen production incidents. So are "consumer lag", "dead letter queue", and "exponential backoff"…
Stop Chasing Exactly-Once. Your Kafka Consumer Just Needs to Not Care
Every few months a team I talk to decides they need exactly-once delivery in Kafka. They read the docs, discover processing.guarantee=exactly_once_v…
How Apache SeaTunnel Handles Kafka Topic Partition Expansion Without a Job Restart
1. Scenario When using an Apache SeaTunnel streaming job to consume data from a Kafka topic, what happens if the topic is expanded from 2 partitions…
The Inbox Transaction Boundary: Getting Event Processing Right in Spring Boot
The Inbox Transaction Boundary: Getting Event Processing Right in Spring Boot One of the hardest parts of building an event-driven system isn't gett…
Did You Know? Kafka Stores Your Messages on Disk 🤔
Most people assume that a high-performance messaging system like Kafka must be keeping everything in memory to stay fast. But here's the truth — eve…
A question on Discord found two bugs in my RAG system. Fixing them found a third.
Every test passed. The tests had the same blind spots as the code. A developer saw Ossian's MCP server in a community showcase and asked two questio…
Resilient Fintech Microservices: High-Throughput with .NET 10 and Apache Kafka
In the fintech sector, data integrity and system reliability are paramount. When managing sensitive operations such as credit products, loan disburs…
Consuming AWS MSK from Azure Databricks over mTLS
Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does…
The compiler was never what you wanted
You have an orders topic on a Kafka cluster, its values encoded with Avro against a schema in the Schema Registry . You want the orders worth more t…
Why I Publish to Kafka Only After the Transaction Commits
The bug that doesn't show up in tests — and what to do about it There is a class of bug in event-driven systems that is almost invisible in developm…
Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency
Event-driven architecture looks simple at first. Your application performs a business operation, publishes an event to Kafka or SQS, and another ser…
Под капотом Kafka: путь сообщения от send() до commit offset. Часть 1
Привет, Хабр! Меня зовут Максим Шуматбаев, я инженер технической поддержки в Arenadata. Эта статья — попытка собрать в одном месте внутренние механи…
«Зачем платформа, если есть Kafka?» Отвечаю честно, включая ту часть, где вы правы
Привет, Хабр! Меня зовут Виктор Овчинников, я руковожу продуктовым направлением «Интеграционная платформа» в «Диасофт». Той самой Digital Q.Integrat…
Большая история крохотного BPMN
Что полезного может делать бизнес-процесс, в котором всего одна задача, и та пользовательская? Что можно рассказать про BPMN-схему, которая поместит…
Two-Layer Idempotency: Why Not Every Settlement System Needs an Outbox
A few weeks ago I published a write-up on inverting the control flow of a settlement API: instead of the endpoint owning window semantics (batching,…
How Kafka Saved a Failing System -The Story of Arjun & Meera
Modern systems run on data in the form of fast, real-time, continuous streams. But what happens when your system grows faster than your architecture…
Инфраструктура данных 2052: что, если у данных больше не будет «прода» и «истории»?
Если мысленно вернуться в 1980-е , аналитика во многом находилась буквально над production Данные появлялись в операционных системах, затем из них с…
Building Fault-Tolerant, Event-Driven Kafka Pipelines in Go: Reliable Reprocessing & Dead Letter Queues
A practical guide to building reliable event-driven systems in Go using Apache Kafka. Learn how to implement tiered retry strategies with delayed re…