Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns
Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation Replacing a legacy ETL platform is…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
Athena read my timestamps as UTC. It didn’t tell me.
Nobody gets an error. Somebody gets blamed. I built a fake metal fab plant to find this kind of bug before a real customer does. Last week it caught…
An AI-Assisted Development Methodology, Earned from Real Incidents
This document is a methodology for building software with AI assistance, compiled from the rules that survived real use on a production data platfor…
The Iceberg Table That Was Never Where We Wrote It
For the past week I've been working on Apache Hop , an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Ic…
Best Data Warehouse Schema Documentation Tools (2026): SQLDBM, dbt, DataHub, Atlan & SchemaSpy Compared
Schema documentation used to be the thing teams promised to do "next sprint" and then never did. In 2026, that excuse doesn't hold up. Warehouses ar…
How AI can investigate a 12 GB export without reading it into a prompt
Imagine asking an AI assistant for the average order value in a 12 GB sales export. It finds an amount column and suggests AVG(line_amount) . The qu…
A bad batch loaded with zero errors. One row added $4.5 million.
I built a small warehouse, fed it a batch of orders with twelve defects planted in it, and watched the load succeed. Every row landed. Summed as loa…
Project Sentinel: One Morning Digest for Every Failure in Our Data Stack
How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…
Moving Your Local Airflow to GCP for under $150 a month
My team runs ML training and prediction pipelines on Vertex AI. For a long time, the thing telling those pipelines when to run was Apache Airflow, i…
Normalizing public job-board data with Python
Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boa…
Never create 1-row ClickHouse parts again: The built-in Buffer Manager.
Day 03 of the WClickHouse Open-Source Engineering Series. You don't need a Redis cluster just to protect ClickHouse from single-event streaming writ…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
We nearly charged our own buyers twice for rows they'd already paid for
Every "watch mode" Actor in our catalog — the ones that poll a source and only deliver new rows since the last run — keeps a baseline of ids it has…
We timed HTTP-only scraping against a headless browser on the same page. It wasn't close.
Every one of our Actors is HTTP-only — no Puppeteer, no Playwright, no headless Chromium in production. We say this is faster and cheaper. This post…
Агент пишет код. Что тогда остаётся инженеру данных?
В работе инженера данных хватает повторяющихся задач: забрать данные из API, проверить структуру, преобразовать поля, записать результат в хранилище…
Your Data Engineering Roadmap Is Probably Too Long
A data engineering roadmap crossed my feed recently. It is thoughtful, detailed, and 571 lines long. The associated Reddit discussion had a predicta…
# Power BI Data Modelling: A Great Path to Great Analysis
Data Modeling in Power BI Introduction At a time when data is rapidly generated by different activities ranging from experiments to business, more o…
5,800 Data Engineers on What Is Actually in Production in 2026
Every year, someone publishes a "State of Data Engineering" survey with 200 respondents, mostly from their own Slack community, and calls it represe…
ELT with Dataform on Google Cloud
This article was originally published on Medium (Google Cloud Community) . 1. Explanation of the use case presented in this article In a previous ar…
Designing a Production-Ready Data Science Pipeline: From Raw Events to Monitored Models
A model is only one artifact in a production data science system. The harder engineering problem is creating a reliable path from changing source da…
Nobody Warns You the Data Engineering Interview Isn't About Data Engineering
I've sat on both sides of the interview table more times than I can count. Here's the thing nobody tells you going in: the interview almost never te…
56 fault-injection tests passed. The one that injected nothing failed.
I was building a tool that detects when data quietly changes meaning — a vendor switching units, a source dropping a field, an undocumented enum app…
Modern Data Lakehouses: How Apache Iceberg Solved the Pitfalls of Hive Metastore
Introduction When a data platform grows from a few gigabytes to terabytes or petabytes, storing the data is only one part of the problem. A typical…
The Hard Truth About Data Engineering Interviews Nobody Tells You
It's 11 PM. You've solved forty LeetCode mediums this month. Your GitHub has a shiny new Airflow project with green checkmarks all the way down. Tom…
5 Habits That Keep a dbt Project From Turning Into a Mess
I once inherited a dbt project where the same "active customer" filter existed in four different models, each written slightly differently. Two of t…
PyArrow, Explained: How Zero-Copy Actually Works, and Why It's Everywhere
Every time data moves between two tools in a modern Python data stack — a DuckDB query result becoming a pandas DataFrame, a Polars frame getting ha…
From Detection to Production: A PII-Safe Pipeline in Python and DuckDB
Part 1 gave you a map: every PII-bearing column, tier-tagged. Part 2 armed you: HMAC for direct identifiers, tokens for what must be reversible, mas…
The hardest label in our dataset is the one where nothing happened
Four times a year, a public company gets on a call with analysts and says what it intends to do next. A margin it expects to reach. A plant it expec…
Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…