Quantified Self: How I Handled 100GB of Messy Biosensor Data with Polars and Dask
We live in the era of the "Quantified Self." Between my Oura ring, Apple Watch, and continuous glucose monitor (CGM), I’m generating more data than…
DevOps
⚑ Report a ProblemLatest DevOps news from Tech News
Startup credits are a moving target — here's how to verify them against an open registry
Every founder eventually collects a folder of "startup credits" links: cloud credits, API allowances, free tiers that scale. The problem is that the…
How to cite an immutable open data registry from an autonomous agent
When an autonomous agent asserts a fact about a company, a product, or an API, the assertion is only as good as its provenance. A number scraped fro…
Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns
Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation Replacing a legacy ETL platform is…
Athena read my timestamps as UTC. It didn’t tell me.
Nobody gets an error. Somebody gets blamed. I built a fake metal fab plant to find this kind of bug before a real customer does. Last week it caught…
Announcing det: - Fast, local-first SQL ETL workspace and pipeline orchestration engine
What if you could build complete, production-grade ETL pipelines in Pure SQL… without Jinja templates, without a database server, and without heavy…
5 SQL window-function traps that show up in data engineering interviews (with runnable DuckDB examples)
Window functions are the most common SQL topic in data engineering interviews, and most candidates know the syntax. The follow-up questions are wher…
An AI-Assisted Development Methodology, Earned from Real Incidents
This document is a methodology for building software with AI assistance, compiled from the rules that survived real use on a production data platfor…
The Iceberg Table That Was Never Where We Wrote It
For the past week I've been working on Apache Hop , an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Ic…
I filtered the FMCSA census (4.5M rows) down to carriers that look real
The FMCSA Company Census is a public file of about 4.5 million US motor carriers, updated daily through the Socrata API (dataset az4n-8mr2 ). It is…
Your agent's worst query is read-only
In April an AI agent deleted a company's production database and HN spent 1,000 comments on it. The details matter: the agent was in Cursor, it foun…
Best Data Warehouse Schema Documentation Tools (2026): SQLDBM, dbt, DataHub, Atlan & SchemaSpy Compared
Schema documentation used to be the thing teams promised to do "next sprint" and then never did. In 2026, that excuse doesn't hold up. Warehouses ar…
Fivetran vs Estuary for Real-Time ETL
Quick answer : Estuary is the better tool for real-time ETL when a consumer needs database changes in seconds: it streams log-based CDC continuously…
Three months of sales vanished, and every dbt run was green
In this article I'll show you how a dbt incremental model that looked perfectly reasonable wiped out three months of finance data, why none of our c…
Why Data Reconciliation Matters in AWS Glue: Finding Missing Records with Anti Joins
A pipeline can be green and still have a data problem. This is one of the things I learned while working with data pipelines. A Glue job can complet…
A bad batch loaded with zero errors. One row added $4.5 million.
I built a small warehouse, fed it a batch of orders with twelve defects planted in it, and watched the load succeed. Every row landed. Summed as loa…
Hello Everyone
Hey! I'm Ayush Shandilya, a software engineer working in data engineering and AI/ML. I'm really interested in understanding how things work under th…
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…
Read Kafka Like a Database
Working with Kafka data on your own terms, using a local index. A search stack usually follows Kafka into a service. But the message you are looking…
Shopify Data in Databricks: A First Look at the New Lakeflow Connect Shopify Connector
Databricks now has a managed Shopify connector in Lakeflow Connect, released in Beta on September 22, 2026. It pulls products, orders, customers, in…
I Gave My Data Pipelines a Memory: Building PipeMind with Hindsight and Groq
The 2 AM problem Every data team has lived this. A pipeline fails in the middle of the night. Somebody gets paged, stares at a stack trace, and thin…
Project Sentinel: One Morning Digest for Every Failure in Our Data Stack
How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…
Moving Your Local Airflow to GCP for under $150 a month
My team runs ML training and prediction pipelines on Vertex AI. For a long time, the thing telling those pipelines when to run was Apache Airflow, i…
Normalizing public job-board data with Python
Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boa…
Cutting a production AWS Glue pipeline's runtime by about half, plus two bugs that taught me more
I spent the last few months tuning a production pipeline on AWS Glue and PySpark : Oracle to S3 ingestion, then packaging, encryption, and transfer…
Has Apache Iceberg Really Solved Vendor Lock-in? Some Thoughts
Original Japanese article : Icebergは本当にベンダーロックインを解消したのか考えてみた Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). With the spread of Apac…
Never create 1-row ClickHouse parts again: The built-in Buffer Manager.
Day 03 of the WClickHouse Open-Source Engineering Series. You don't need a Redis cluster just to protect ClickHouse from single-event streaming writ…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
Mengenal Data Engineering: Pahlawan di Balik Layar Dunia Data - 23:08
Pernahkah Anda bertanya-tanya bagaimana perusahaan raksasa seperti Netflix atau Google mengelola data yang jumlahnya triliunan gigabyte setiap harin…