Hosting an Entire Open Lakehouse on Port 8080
Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost ne…
DevOps
⚑ Report a ProblemLatest DevOps news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
cloud
database
devchallenge
devops
docker
javascript
kubernetes
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
All About Iceberg's File Format API: simplifying the engine/format interface and Parquet's multimodal future
What is the file format API? Apache Iceberg traditionally exposed its data files through three libraries, one each for Parquet, Avro and ORC, and un…
AUTO_CDC: Turning an event log into current state with Spark Declarative Pipelines
The question an event log can't answer cheaply The dataset I use for most of my pipeline demos is a simulated ghost kitchen (a delivery-only restaur…
The Iceberg Table That Was Never Where We Wrote It
For the past week I've been working on Apache Hop , an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Ic…
Has Apache Iceberg Really Solved Vendor Lock-in? Some Thoughts
Original Japanese article : Icebergは本当にベンダーロックインを解消したのか考えてみた Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). With the spread of Apac…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
The Snowflake Iceberg Pivot: Why Your Data Warehouse Should Be a Storage Engine
Ninety percent of the "performance" you pay for in Snowflake is actually just a tax on your inability to move your data. If you’re like most enginee…
Four Iceberg Tools, Three Agent Frameworks: What Ports, and What Doesn't
This article is a step by step build of one small Apache Iceberg data agent, written three times: in Google ADK, in AWS Strands and in Microsoft Age…
Four Iceberg Tools, Three Agent Frameworks: What Ports, and What Doesn't
This article is a step by step build of one small Apache Iceberg data agent, written three times: in Google ADK, in AWS Strands and in Microsoft Age…
Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…
Как я написал свой мигратор для ClickHouse — и почему он до сих пор жив
В 2020-м мне понадобилось версионировать схему ClickHouse в CI/CD, а готового инструмента под Go с поддержкой ClickHouse не нашлось ни одного. Пришл…
The query succeeded. Which table state did it read?
Why two successful queries can count different rows when they resolve different Delta or Iceberg table states. Two queries return different counts f…
One Histogram, Four Engines: Fixing the Statistics Silo Problem
TL;DR — DuckDB, DataFusion, Polars, and Postgres each compute and store table statistics their own way, so a histogram built in your ELT pipeline is…
Why Athena/Iceberg Tends to Make Code the Spec
Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole…
Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables
Original Japanese article : Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する Introduction I'm Aki, an AWS Community Builder ( @jitepen…
AI Enrichment Pipeline: From Sample Classification to 100K-File Metadata Search with Bedrock and OpenSearch NextGen
Quick Recap: What We Built in Part 1 In Part 1 , we built a metadata catalog on Apache Iceberg (S3 Tables) that makes unstructured files on FSx for…
From Hours to Seconds: An AI-Powered Metadata Catalog for Unstructured Data on FSx for ONTAP
What Works Now vs What Requires Validation This article separates verified AWS-native capabilities from cross-platform paths that still require vali…
Только Сигма выбирают Delta Lake
Привет, Хабр! Меня зовут Дмитрий Кравчук, я занимаюсь всем, что связано с данными в блоке AI&ML MAGNIT TECH. Расскажу про фундамент прибыльных п…
Rethinking Lakehouse Architecture Through Data Ownership: AWS vs. Snowflake
Original Japanese article : データの主導権から考えるAWSとSnowflakeのレイクハウスアーキテクチャ Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). When designing a…
Load PostgreSQL into Apache Iceberg with Sling
Introduction Apache Iceberg is the table format that turns a pile of Parquet files in object storage into something that behaves like a warehouse ta…
When does Iceberg beat Parquet+projection on AWS Glue, and when doesn't ?
Why this project I built this repo because I didn't have one of this kind yet and, having worked on data ingestion with Glue for a while, I wanted t…
Ursa — a new Diskless Lakestream engine for Kafka
The future of Kafka is diskless topics + native Apache Iceberg / Delta Lake integration Intro Ursa is a relatively new engine that's being fitted in…