Open Source — Tech News

An Idiot's Guide to Spark Connect Clients

Spark Connect is a new concept for many of us, especially given the long history of how managing Spark clusters has traditionally worked. Spark Conn…

spark sparkconnect pyspark

Spark is moving its centre of gravity out of the JVM. Spark Connect, introduced in 3.4, lets a client drive Spark over gRPC without running a JVM of…

spark pyspark sparkconnect streaming

Arrow's real contribution to PySpark isn't speed at any one API but a shared memory format at every boundary between the JVM and Python; each releas…

spark pyspark arrow history

When I benchmarked PySpark's UDF types earlier this year, I ran the same three workloads (trivial arithmetic, a string transform, a normal CDF) at f…

spark pyspark performance arrow

The benchmark that tested the wrong Arrow Earlier this year I built a benchmark to answer a question that comes up constantly: if you need custom Py…

spark pyspark arrow benchmarks

Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or…

databricks pyspark python dataengineering

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On pa…