Tech news from the best sources
I thought this would be a simple task. I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a…
I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memor…
Recently, I reviewed an Airflow DAG where each task submits a single Spark job. I found this configuration: execution_timeout_minutes : 60 spark-job…
In distributed data processing systems such as Apache Spark, joins are among the most expensive operations. The strategy used to join datasets can s…
If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash w…