Building My First End-to-End ETL Pipeline with Airflow, BigQuery, and Docker
Recently, I completed my first full Data Engineering project: building an end-to-end ETL pipeline using real-world Australian weather data spanning 10…
DevOps
⚑ Report a ProblemLatest DevOps news from Tech News
All topics
agents
ai
api
architecture
automation
aws
beginners
career
cloud
database
devchallenge
devops
docker
gemma
javascript
kubernetes
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
tutorial
typescript
webdev
Why Metadata-Driven ETL Frameworks Scale Better Than Hardcoded Pipelines — and Where They Don't
Over the years, I've seen many data platforms start with good intentions. A few scripts are created to move data from one system to another, and every…
A torrent search engine,for actually active torrents!
The core paradox of modern BitTorrent: how do you search without a central index? Searching needs metadata, an address that you can query. BitTorrent …
Extract data from Databases into DuckLake
Introduction In the evolving landscape of data engineering, DuckLake is emerging as a powerful solution for building data lakes with ACID transactions…
ETL Pipeline: Fetching Real-Time News Data with Python and Postgres
The best way to actually understand data engineering is to build something that breaks, fix it, and watch it successfully run. In this article, we bui…
Replicate MySQL to ClickHouse with Sling
Introduction ClickHouse is a columnar OLAP database. It runs aggregate queries across billions of rows in seconds. MySQL is what most apps run on for …
Load PostgreSQL into Apache Iceberg with Sling
Introduction Apache Iceberg is the table format that turns a pile of Parquet files in object storage into something that behaves like a warehouse tabl…
Миграция с Greenplum. Эпизод I: Атака клонов и спасение на звёздных камнях
В мае 2024 года Broadcom заархивировал публичный репозиторий Greenplum: последний коммит остался на месте, дальнейшая разработка ушла в закрытый репоз…