Tech news from the best sources
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…
Databricks now has a managed Shopify connector in Lakeflow Connect, released in Beta on September 22, 2026. It pulls products, orders, customers, in…
I Gave Every Pull Request Its Own Database Git-style branching for Postgres, wired into Jenkins, with a real repo you can run. Every team has a dev…
Why I chose this topic: I spent three days last quarter fixing a "simple" metadata sync that locked 200 production jobs out of their own tables beca…
The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told…
It’s Sunday, 2:14 AM. The PagerDuty alert hits my phone with that specific, jarring frequency that makes your stomach drop before you’ve even opened…
"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer…
Databricks launched its new Adaptive Instructed-Retriever this week to help businesses refine their AI-driven search capabilities. The model focuses…
Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does…
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…
Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL…
I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On pa…
Databricks is acquiring the startup Electric to integrate WebAssembly-based Postgres databases directly into application environments for an undiscl…
Every Databricks pitch deck has the same three-layer diagram: bronze for raw data, silver for cleaned data, gold for business-ready tables. The diag…
A follow-up to Part 2: Databricks and FSx for ONTAP S3 Access Points in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some resu…
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand how Azure Databricks projects are…
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand two related but different topics:…
The pitch for agentic AI on enterprise data is genuinely appealing: instead of a chatbot that answers one question at a time, you get something that…
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand common Delta Lake and Spark table…
Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but…
Databricks is expanding its Lakebridge toolkit by introducing an agentic code conversion feature designed to help organizations migrate from legacy…
Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fail…
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffl…
For most of the last decade, the honest answer to "should we standardize on Databricks or on the Microsoft data platform?" was: pick one and live wi…
As organisations move to the Databricks Lakehouse Platform on AWS, two priorities consistently surface at the top of every architecture conversation…
The problem We had a reference dataset in Databricks with 1B+ rows . Users needed to look up a handful of those rows from a Salesforce record page.…
📚 Series: Databricks on AWS (Part 5) Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Cl…
📚 Series: Databricks on AWS (Part 4) Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Cl…
📚 Series: Databricks on AWS (Part 3) Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Cl…
📚 Series: Databricks on AWS (Part 2) Building a Databricks AI Platform on AWS RBAC with Function-Role Groups ← you are here Compute Governance: Pool…