Apache Airflow on 73,265 observed hosts: the scheduler that holds every connection string
Apache Airflow on 73,265 observed hosts: the scheduler that holds every connection string A ZoomEye fingerprint query for app="Apache Airflow" ( res…
Последние новости Web от Tech News
Все темы
AI
agents
ai
api
architecture
automation
backend
beginners
career
database
devchallenge
devops
discuss
javascript
linux
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Project Sentinel: One Morning Digest for Every Failure in Our Data Stack
How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…
73,105 Airflow instances, 11,673 SonarQube and 33,837 Nexus: fingerprinting the build and quality toolchain
73,105 Airflow instances, 11,673 SonarQube and 33,837 Nexus: fingerprinting the build and quality toolchain The counts Three products that sit in th…
Stop trying to make Airflow work for Medallion pipelines
The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told…
Understanding Apache Airflow 3.3: Key Improvements, Features Every Data Engineer Should Know, and Real-World Use Cases.
It is relatively straightforward to design a data pipeline that works under the perfect conditions of responsive APIs, flawless networks and where c…
Installing Apache Airflow on Ubuntu 24.04
Apache Airflow is an open-source workflow management platform that manages data pipelines and automates workflows such as Extract, Transform, and Lo…
Understanding Apache Airflow 3.3: Key Improvements, Features Every Data Engineer Should Know, and Real-World Use Cases
Airflow 3.3.0 shipped on 6 July 2026, with 3.3.1 following about five weeks later. It's a much smaller release than 3.0 was, and that's fine. Nothin…
How to integrate Apache Airflow with OpenLineage for end-to-end traceability
How to integrate Apache Airflow with OpenLineage for end-to-end traceability By the end of this walkthrough, every DAG run in your Airflow instance…
🕳️ HomeLab Chronicles: Episode 9 - Tunnel Vision
Hey all 👋 State of the lab: two nodes, Flux in charge, Airflow alive, backups nightly, RBAC banished to therapy. The UI works from anywhere in the h…
🌪️ HomeLab Chronicles: Episode 4 - Turbulence
Hey all 👋 Sit tight! Because this is a little lengthy one. Last episode I closed by announcing that I would be adding two more nodes and bringing in…
10 инсайтов, которые я нашел, когда автоматизировал конкурентную аналитику
Привет, Хабр! На связи Влад Михайлов, Marketing Analyst в Garage Eight. Не знаю, замечали ли вы, но последние несколько лет открытых данных о мобиль…
Databricks Workflows vs Airflow vs Dagster: Picking an Orchestrator
Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fail…
CI/CD для данных и моделей на Airflow: как мы деплоим прогноз спроса в «Магните»
Артефакт нашей сборки — не бинарник, а код вместе с терабайтами рассчитанных таблиц. Рассказываем, как из штатных фич Airflow, Spark и Delta Lake у…
Airflow: A Beginner's Guide
Introduction: I Built a Pipeline. Now What? In the previous article Understanding ETL: A Chaotic Introduction , I built a simple pipeline, and after…
Kafka for Data Engineers: Core Concepts, KRaft, and the Patterns That Actually Work
If your Kafka Docker Compose still has a ZooKeeper service in it, your setup is already legacy. As of Kafka 4.0 (released March 2025), ZooKeeper is…
My 10-Minute Airflow Pitch Approach
"Good Morning, can you check if the pipelines ran successfully?". A sentence that probably many of us know or heard a few times before. But, wouldn'…
Код как документация: как мы строим самодокументируемые витрины данных в Почте Mail
В аналитике больших данных есть старая проблема: код ETL-витрин живет своей жизнью, а документация — своей. Изменяешь логику, забываешь обновить опи…
Код как документация: как мы строим самодокументируемые витрины данных в Почте Mail
В аналитике больших данных есть старая проблема: код ETL-витрин живет своей жизнью, а документация — своей. Изменяешь логику, забываешь обновить опи…
We Upgraded Airflow 2.8 to 3.1 on Kubernetes. Here Is What Actually Changed
We recently finished upgrading a production Airflow instance from 2.8 to 3.1 running on Amazon EKS . The whole thing took about 6 weeks from plannin…
Building Your First Airflow DAG: Extracting Stock Data with Massive
When stepping into the world of data engineering, Apache Airflow is likely one of the first tools you will encounter. It is the industry standard fo…
A Beginners Guide to Apache Airflow
Introduction In data engineering we build data pipelines using approaches such as ETL(extract, transform, load) and ELT(extract, load, transform). T…