Shopify Data in Databricks: A First Look at the New Lakeflow Connect Shopify Connector
Databricks now has a managed Shopify connector in Lakeflow Connect, released in Beta on September 22, 2026. It pulls products, orders, customers, in…
Architecture
⚑ Сообщить о проблемеПоследние новости Architecture от Tech News
Все темы
agents
ai
api
architecture
automation
aws
backend
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
news
opensource
performance
productivity
programming
python
react
security
showdev
softwareengineering
systemdesign
tutorial
typescript
webdev
I Gave Every Pull Request Its Own Database
I Gave Every Pull Request Its Own Database Git-style branching for Postgres, wired into Jenkins, with a real repo you can run. Every team has a dev…
Stop trying to make Airflow work for Medallion pipelines
The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told…
How Databricks Serverless Compute Cost My Team $14k in One Weekend
It’s Sunday, 2:14 AM. The PagerDuty alert hits my phone with that specific, jarring frequency that makes your stomach drop before you’ve even opened…
Databricks Architecture Explained: Control Plane, Compute Plane, Delta Lake and Unity Catalog
"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer…
Databricks Adaptive Instructed-Retriever model cuts AI costs
Databricks launched its new Adaptive Instructed-Retriever this week to help businesses refine their AI-driven search capabilities. The model focuses…
Consuming AWS MSK from Azure Databricks over mTLS
Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does…
Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…
Azure Synapse Databricks: A Migration Playbook That Survives Production
Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL…
Databricks buys Electric to embed local Postgres in AI agents
Databricks is acquiring the startup Electric to integrate WebAssembly-based Postgres databases directly into application environments for an undiscl…
Medallion architecture on Databricks: what actually matters
Every Databricks pitch deck has the same three-layer diagram: bronze for raw data, silver for cleaned data, gold for business-ready tables. The diag…
Using Databricks FILE type against data on NAS: what works today with FSx for ONTAP S3 Access Points
A follow-up to Part 2: Databricks and FSx for ONTAP S3 Access Points in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some resu…
Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffl…
One Copy, Two Engines: Architecting a Governed Lakehouse Across Fabric and Azure Databricks
For most of the last decade, the honest answer to "should we standardize on Databricks or on the Microsoft data platform?" was: pick one and live wi…
🔐 Storage Credentials in Databricks Unity Catalog (AWS) — The Security Backbone You Can't Ignore
As organisations move to the Databricks Lakehouse Platform on AWS, two priorities consistently surface at the top of every architecture conversation…
How to Query Databricks from Salesforce Apex (Without Copying a Billion Rows)
The problem We had a reference dataset in Databricks with 1B+ rows . Users needed to look up a handful of those rows from a Salesforce record page.…
[Databricks on AWS #0] The Target Architecture: Isolating Prod, Dev, and Sandbox with Unity Catalog
📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Ro…
Azure Databricks vs Microsoft Fabric: An Honest Guide to When to Use What
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta…
Azure Databricks for MLOps and Feature Engineering at Scale with Apache Spark, Delta Lake, and MLflow
Raw data doesn't win model competitions. Features do. And when your raw data is tens of billions of rows sitting across multiple sources, you can't…
How I Built a Databricks AI Agent with No Custom Tables (OpenAI Agents SDK + Gradio)
Most Databricks agent tutorials start with "set up Unity Catalog and Vector Search first." This one doesn't. I built a fully working conversational…
Real-Time AI Feature Engineering with Spark Structured Streaming and Databricks Feature Store
Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark St…
How I debugged a Delta Lake DESCRIBE HISTORY timeout (and what's actually causing it)
If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash w…