Project Sentinel: One Morning Digest for Every Failure in Our Data Stack
How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…
Architecture
⚑ Report a ProblemLatest Architecture news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
news
opensource
performance
productivity
programming
python
react
security
showdev
softwareengineering
systemdesign
tutorial
typescript
webdev
Stop trying to make Airflow work for Medallion pipelines
The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told…
Understanding Apache Airflow 3.3: Key Improvements, Features Every Data Engineer Should Know, and Real-World Use Cases
Airflow 3.3.0 shipped on 6 July 2026, with 3.3.1 following about five weeks later. It's a much smaller release than 3.0 was, and that's fine. Nothin…
Свой инструмент для бенчмаркинга ИИ-агентов: архитектура, надёжность и интеграция с Airflow
Всем привет! Мы создаём GraphRAG-систему и нам постоянно приходится тестировать новые гипотезы: менять подходы к поиску по графу, обработку контекст…
Kafka for Data Engineers: Core Concepts, KRaft, and the Patterns That Actually Work
If your Kafka Docker Compose still has a ZooKeeper service in it, your setup is already legacy. As of Kafka 4.0 (released March 2025), ZooKeeper is…
We Upgraded Airflow 2.8 to 3.1 on Kubernetes. Here Is What Actually Changed
We recently finished upgrading a production Airflow instance from 2.8 to 3.1 running on Amazon EKS . The whole thing took about 6 weeks from plannin…