An Idiot's Guide to Spark Connect Clients
Spark Connect is a new concept for many of us, especially given the long history of how managing Spark clusters has traditionally worked. Spark Conn…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
Spark's Modules, Seen Through Spark Connect
Spark is moving its centre of gravity out of the JVM. Spark Connect, introduced in 3.4, lets a client drive Spark over gRPC without running a JVM of…
Explaining the Python-JVM Boundary in Spark, and What Arrow Changed
Arrow's real contribution to PySpark isn't speed at any one API but a shared memory format at every boundary between the JVM and Python; each releas…
Most PySpark Tuning Advice Starts in the Wrong Place
When I benchmarked PySpark's UDF types earlier this year, I ran the same three workloads (trivial arithmetic, a string transform, a normal CDF) at f…
Stop Blaming Serialization for Slow Python UDFs
The benchmark that tested the wrong Arrow Earlier this year I built a benchmark to answer a question that comes up constantly: if you need custom Py…
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…
95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.
I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On pa…