Group exact brand labels while keeping identity resolution explicit
I maintain this Actor. This is an AI-assisted technical walkthrough checked against the installed source contract. Grouping by brand name can be use…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
How to Scrape Telegram Channels and Groups
Key Takeaways Telegram provides a free official API. So, the main challenges lie in following privacy regulations, adapting to MTProto, and managing…
Snowflake Meets Collibra: A Data Engineer Interviews a Governance Expert
Collibra helps a company know its data: what it means, where it came from, if it is correct, and who can use it. This interview explains it in plain…
Quantified Self: How I Handled 100GB of Messy Biosensor Data with Polars and Dask
We live in the era of the "Quantified Self." Between my Oura ring, Apple Watch, and continuous glucose monitor (CGM), I’m generating more data than…
Startup credits are a moving target — here's how to verify them against an open registry
Every founder eventually collects a folder of "startup credits" links: cloud credits, API allowances, free tiers that scale. The problem is that the…
How to cite an immutable open data registry from an autonomous agent
When an autonomous agent asserts a fact about a company, a product, or an API, the assertion is only as good as its provenance. A number scraped fro…
Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns
Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation Replacing a legacy ETL platform is…
Athena read my timestamps as UTC. It didn’t tell me.
Nobody gets an error. Somebody gets blamed. I built a fake metal fab plant to find this kind of bug before a real customer does. Last week it caught…
Announcing det: - Fast, local-first SQL ETL workspace and pipeline orchestration engine
What if you could build complete, production-grade ETL pipelines in Pure SQL… without Jinja templates, without a database server, and without heavy…
5 SQL window-function traps that show up in data engineering interviews (with runnable DuckDB examples)
Window functions are the most common SQL topic in data engineering interviews, and most candidates know the syntax. The follow-up questions are wher…
An AI-Assisted Development Methodology, Earned from Real Incidents
This document is a methodology for building software with AI assistance, compiled from the rules that survived real use on a production data platfor…
The Iceberg Table That Was Never Where We Wrote It
For the past week I've been working on Apache Hop , an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Ic…
I filtered the FMCSA census (4.5M rows) down to carriers that look real
The FMCSA Company Census is a public file of about 4.5 million US motor carriers, updated daily through the Socrata API (dataset az4n-8mr2 ). It is…
Your agent's worst query is read-only
In April an AI agent deleted a company's production database and HN spent 1,000 comments on it. The details matter: the agent was in Cursor, it foun…
Best Data Warehouse Schema Documentation Tools (2026): SQLDBM, dbt, DataHub, Atlan & SchemaSpy Compared
Schema documentation used to be the thing teams promised to do "next sprint" and then never did. In 2026, that excuse doesn't hold up. Warehouses ar…
Fivetran vs Estuary for Real-Time ETL
Quick answer : Estuary is the better tool for real-time ETL when a consumer needs database changes in seconds: it streams log-based CDC continuously…
Three months of sales vanished, and every dbt run was green
In this article I'll show you how a dbt incremental model that looked perfectly reasonable wiped out three months of finance data, why none of our c…
Every Split Leaves a Fingerprint: Checking Corporate Actions Against Raw Prices
Python for programmers, prompts for data analysts. Part 33 found a crash that never happened: Amazon's raw price fell 95% overnight because of a 20-…
How AI can investigate a 12 GB export without reading it into a prompt
Imagine asking an AI assistant for the average order value in a 12 GB sales export. It finds an amount column and suggests AVG(line_amount) . The qu…
Why Data Reconciliation Matters in AWS Glue: Finding Missing Records with Anti Joins
A pipeline can be green and still have a data problem. This is one of the things I learned while working with data pipelines. A Glue job can complet…
A bad batch loaded with zero errors. One row added $4.5 million.
I built a small warehouse, fed it a batch of orders with twelve defects planted in it, and watched the load succeed. Every row landed. Summed as loa…
Hello Everyone
Hey! I'm Ayush Shandilya, a software engineer working in data engineering and AI/ML. I'm really interested in understanding how things work under th…
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…
Read Kafka Like a Database
Working with Kafka data on your own terms, using a local index. A search stack usually follows Kafka into a service. But the message you are looking…
Shopify Data in Databricks: A First Look at the New Lakeflow Connect Shopify Connector
Databricks now has a managed Shopify connector in Lakeflow Connect, released in Beta on September 22, 2026. It pulls products, orders, customers, in…
I Gave My Data Pipelines a Memory: Building PipeMind with Hindsight and Groq
The 2 AM problem Every data team has lived this. A pipeline fails in the middle of the night. Somebody gets paged, stares at a stack trace, and thin…
Project Sentinel: One Morning Digest for Every Failure in Our Data Stack
How we collect health metadata from Airflow, Glue, dbt and Tableau into S3, serve it through a thin API, and let an AI agent write the daily report.…
Moving Your Local Airflow to GCP for under $150 a month
My team runs ML training and prediction pipelines on Vertex AI. For a long time, the thing telling those pipelines when to run was Apache Airflow, i…
Data Science vs. Data Engineering: Mana Jalur Karier yang Tepat untuk Kamu? - 20:50
Di era digital saat ini, data sering disebut sebagai 'minyak baru'. Namun, layaknya minyak mentah, data tidak akan berguna tanpa proses pengolahan y…
Normalizing public job-board data with Python
Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boa…