Мы перестали смотреть на дашборды расходов и добавили стоимость прямо в интерфейс Databricks
А что если вместо дашборда по затратам Databricks добавить их прямо в интерфейс. С этой идеи начался наш мини проект, который решал сначала нашу лок…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
- игры
AI
News
Tech
agents
ai
api
architecture
automation
backend
beginners
career
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…
Shopify Data in Databricks: A First Look at the New Lakeflow Connect Shopify Connector
Databricks now has a managed Shopify connector in Lakeflow Connect, released in Beta on September 22, 2026. It pulls products, orders, customers, in…
I Gave Every Pull Request Its Own Database
I Gave Every Pull Request Its Own Database Git-style branching for Postgres, wired into Jenkins, with a real repo you can run. Every team has a dev…
Don't Let Unity Catalog Migration Become Your Resume Generating Event
Why I chose this topic: I spent three days last quarter fixing a "simple" metadata sync that locked 200 production jobs out of their own tables beca…
Stop trying to make Airflow work for Medallion pipelines
The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told…
How Databricks Serverless Compute Cost My Team $14k in One Weekend
It’s Sunday, 2:14 AM. The PagerDuty alert hits my phone with that specific, jarring frequency that makes your stomach drop before you’ve even opened…
Databricks Architecture Explained: Control Plane, Compute Plane, Delta Lake and Unity Catalog
"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer…
Databricks Adaptive Instructed-Retriever model cuts AI costs
Databricks launched its new Adaptive Instructed-Retriever this week to help businesses refine their AI-driven search capabilities. The model focuses…
Consuming AWS MSK from Azure Databricks over mTLS
Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does…
Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…
Azure Synapse Databricks: A Migration Playbook That Survives Production
Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL…
95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.
I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On pa…
[Перевод] Как перенести сертификаты и значки Databricks при смене работодателя
Рынок ИТ нестабилен, и люди могут менять работодателей. Ваша предыдущая компания была партнером Databricks, и у вас был аккаунт в Databricks Partner…
Databricks buys Electric to embed local Postgres in AI agents
Databricks is acquiring the startup Electric to integrate WebAssembly-based Postgres databases directly into application environments for an undiscl…
Medallion architecture on Databricks: what actually matters
Every Databricks pitch deck has the same three-layer diagram: bronze for raw data, silver for cleaned data, gold for business-ready tables. The diag…
Using Databricks FILE type against data on NAS: what works today with FSx for ONTAP S3 Access Points
A follow-up to Part 2: Databricks and FSx for ONTAP S3 Access Points in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some resu…
DP-750: Databricks Asset Bundles, Git, and ALM Explained and with Real Exam Questions
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand how Azure Databricks projects are…
DP-750: Slowly Changing Dimensions and Data Quality Expectations Explained and with Real Exam Questions
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand two related but different topics:…
Agentic AI and big data: what your platform needs before agents are useful
The pitch for agentic AI on enterprise data is genuinely appealing: instead of a chatbot that answers one question at a time, you get something that…
DP-750: Delta Lake Table Operations Explained and with Real Exam Questions
When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand common Delta Lake and Spark table…
Snowflake to Databricks: what the migration actually costs you
Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but…
Databricks launches AI agent for legacy SQL migration
Databricks is expanding its Lakebridge toolkit by introducing an agentic code conversion feature designed to help organizations migrate from legacy…
Databricks Workflows vs Airflow vs Dagster: Picking an Orchestrator
Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fail…
Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffl…
One Copy, Two Engines: Architecting a Governed Lakehouse Across Fabric and Azure Databricks
For most of the last decade, the honest answer to "should we standardize on Databricks or on the Microsoft data platform?" was: pick one and live wi…
🔐 Storage Credentials in Databricks Unity Catalog (AWS) — The Security Backbone You Can't Ignore
As organisations move to the Databricks Lakehouse Platform on AWS, two priorities consistently surface at the top of every architecture conversation…
How to Query Databricks from Salesforce Apex (Without Copying a Billion Rows)
The problem We had a reference dataset in Databricks with 1B+ rows . Users needed to look up a handful of those rows from a Salesforce record page.…
Databricks Data and AI Summit 2026. Моя первая поездка в США
Недавно мне удалось посетить Data + AI Summit в Сан-Франциско в качестве Databricks MVP . Крупнейшую конференцию Databricks, посвященную данным, иск…
[Databricks on AWS #5] Fixing Databricks BOOTSTRAP_TIMEOUT with AWS PrivateLink: Control Plane Over the Backbone, Zero New Subnets
📚 Series: Databricks on AWS (Part 5) Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Cl…