Hosting an Entire Open Lakehouse on Port 8080
Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost ne…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
An Idiot's Guide to Spark Connect Clients
Spark Connect is a new concept for many of us, especially given the long history of how managing Spark clusters has traditionally worked. Spark Conn…
Spark's Modules, Seen Through Spark Connect
Spark is moving its centre of gravity out of the JVM. Spark Connect, introduced in 3.4, lets a client drive Spark over gRPC without running a JVM of…
AUTO_CDC: Turning an event log into current state with Spark Declarative Pipelines
The question an event log can't answer cheaply The dataset I use for most of my pipeline demos is a simulated ghost kitchen (a delivery-only restaur…
Explaining the Python-JVM Boundary in Spark, and What Arrow Changed
Arrow's real contribution to PySpark isn't speed at any one API but a shared memory format at every boundary between the JVM and Python; each releas…
Most PySpark Tuning Advice Starts in the Wrong Place
When I benchmarked PySpark's UDF types earlier this year, I ran the same three workloads (trivial arithmetic, a string transform, a normal CDF) at f…
Stop Blaming Serialization for Slow Python UDFs
The benchmark that tested the wrong Arrow Earlier this year I built a benchmark to answer a question that comes up constantly: if you need custom Py…
Cutting a production AWS Glue pipeline's runtime by about half, plus two bugs that taught me more
I spent the last few months tuning a production pipeline on AWS Glue and PySpark : Oracle to S3 ingestion, then packaging, encryption, and transfer…
Don't Let Unity Catalog Migration Become Your Resume Generating Event
Why I chose this topic: I spent three days last quarter fixing a "simple" metadata sync that locked 200 production jobs out of their own tables beca…
Why are you still debugging Spark on your host machine?
Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly $12,000 in cloud compute credits. A developer ha…
Microsoft Fabric Now Encrypts Spark Temporary Data with Customer-Managed Keys;What It Actually Means
Microsoft Fabric has extended customer-managed key (CMK) encryption to cover temporary data generated during Spark job execution. This is not a mino…
Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffl…
Azure Databricks vs Microsoft Fabric: An Honest Guide to When to Use What
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta…
Azure Databricks for MLOps and Feature Engineering at Scale with Apache Spark, Delta Lake, and MLflow
Raw data doesn't win model competitions. Features do. And when your raw data is tens of billions of rows sitting across multiple sources, you can't…
Real-Time AI Feature Engineering with Spark Structured Streaming and Databricks Feature Store
Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark St…
Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine
A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon ve…
Google is pitching an AI agent ecosystem to consumers who may not buy it
The AI agents are coming. A lot of them.
Why My Spark Container Keeps Exiting — Docker PID 1 and the Daemon Trap
I spent an embarrassing amount of time staring at my terminal, watching Spark containers start and immediately die. Three different attempts, three…