Testing & QA — Tech News

Project Sentinel: One Morning Digest for Every Failure in Our Data Stack

How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…

dataengineering airflow aws ai

Airflow 3.3.0 shipped on 6 July 2026, with 3.3.1 following about five weeks later. It's a much smaller release than 3.0 was, and that's fine. Nothin…

How to integrate Apache Airflow with OpenLineage for end-to-end traceability By the end of this walkthrough, every DAG run in your Airflow instance…

Hey all 👋 Sit tight! Because this is a little lengthy one. Last episode I closed by announcing that I would be adding two more nodes and bringing in…

airflow kubernetes helm devops

Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fail…

Работая единственным QA в команде, в обязанности которого активно входит мониторинг, я столкнулась с проблемами тайм-менеджмента: очень много времен…

Всем привет! Мы создаём GraphRAG-систему и нам постоянно приходится тестировать новые гипотезы: менять подходы к поиску по графу, обработку контекст…

If your Kafka Docker Compose still has a ZooKeeper service in it, your setup is already legacy. As of Kafka 4.0 (released March 2025), ZooKeeper is…

dataengineering kafka airflow streaming

We recently finished upgrading a production Airflow instance from 2.8 to 3.1 running on Amazon EKS . The whole thing took about 6 weeks from plannin…

airflow dataengineering kubernetes eks

Introduction In data engineering we build data pipelines using approaches such as ETL(extract, transform, load) and ELT(extract, load, transform). T…