Explaining the Python-JVM Boundary in Spark, and What Arrow Changed
Arrow's real contribution to PySpark isn't speed at any one API but a shared memory format at every boundary between the JVM and Python; each releas…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
Most PySpark Tuning Advice Starts in the Wrong Place
When I benchmarked PySpark's UDF types earlier this year, I ran the same three workloads (trivial arithmetic, a string transform, a normal CDF) at f…
Stop Blaming Serialization for Slow Python UDFs
The benchmark that tested the wrong Arrow Earlier this year I built a benchmark to answer a question that comes up constantly: if you need custom Py…