Tech News — Latest News

All topics - игры AI Gear News Tech agents ai api architecture automation beginners career claude devchallenge devops javascript llm machinelearning mcp opensource performance productivity programming python react security showdev tutorial typescript webdev

All EN RU

Aligning Timeouts in Distributed Orchestration: Why Equal Airflow and Spark Limits Lead to Race Conditions

Recently, I reviewed an Airflow DAG where each task submits a single Spark job. I found this configuration: execution_timeout_minutes : 60 spark-job-t…

dataengineering apacheairflow apachespark dataplatform

Broadcast Joins vs. Sort-Merge Joins: Choosing the Right Join Strategy in Apache Spark

In distributed data processing systems such as Apache Spark, joins are among the most expensive operations. The strategy used to join datasets can sig…

apachespark sql joins

How I debugged a Delta Lake DESCRIBE HISTORY timeout (and what's actually causing it)

If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash wit…

dataengin databricks apachespark deltalake