Hosting an Entire Open Lakehouse on Port 8080
Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost ne…
Architecture
⚑ Report a ProblemLatest Architecture news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
news
opensource
performance
productivity
programming
python
react
security
showdev
softwareengineering
systemdesign
tutorial
typescript
webdev
Cutting a production AWS Glue pipeline's runtime by about half, plus two bugs that taught me more
I spent the last few months tuning a production pipeline on AWS Glue and PySpark : Oracle to S3 ingestion, then packaging, encryption, and transfer…
Microsoft Fabric Now Encrypts Spark Temporary Data with Customer-Managed Keys;What It Actually Means
Microsoft Fabric has extended customer-managed key (CMK) encryption to cover temporary data generated during Spark job execution. This is not a mino…
Всё что нужно знать про DuckLake
Когда вышел DuckDB 1.0, я написал статью Всё что нужно знать про DuckDB . Теперь вышел DuckLake v1.0 — LakeHouse-формат из той же экосистемы. Это от…
[Перевод] Работа с AI/ML-нагрузками в Kubernetes: плагин Headlamp для Kubeflow
Kubernetes незаметно стал платформой по умолчанию для ИИ и машинного обучения. Запускаете ли вы серверы notebook-ов для дата-сайентистов, планируете…
Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffl…
Azure Databricks vs Microsoft Fabric: An Honest Guide to When to Use What
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta…
Azure Databricks for MLOps and Feature Engineering at Scale with Apache Spark, Delta Lake, and MLflow
Raw data doesn't win model competitions. Features do. And when your raw data is tens of billions of rows sitting across multiple sources, you can't…
Real-Time AI Feature Engineering with Spark Structured Streaming and Databricks Feature Store
Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark St…