Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to Rest-Based Architecture
Slack modernized its data platform by replacing SSH based execution in Amazon EMR pipelines with a REST driven orchestration layer called Quarry. The …
DevOps
⚑ Report a ProblemLatest DevOps news from Tech News
All topics
agents
ai
api
architecture
automation
aws
beginners
career
cloud
database
devchallenge
devops
docker
gemma
javascript
kubernetes
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
tutorial
typescript
webdev
Article: Two Misconfigurations That Caused Spark OOM Failures on Kubernetes
After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=t…
Article: From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline
This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why re…