Hosting an Entire Open Lakehouse on Port 8080
Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost ne…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
All About Iceberg's File Format API: simplifying the engine/format interface and Parquet's multimodal future
What is the file format API? Apache Iceberg traditionally exposed its data files through three libraries, one each for Parquet, Avro and ORC, and un…
AUTO_CDC: Turning an event log into current state with Spark Declarative Pipelines
The question an event log can't answer cheaply The dataset I use for most of my pipeline demos is a simulated ghost kitchen (a delivery-only restaur…
The Iceberg Table That Was Never Where We Wrote It
For the past week I've been working on Apache Hop , an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Ic…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
The Snowflake Iceberg Pivot: Why Your Data Warehouse Should Be a Storage Engine
Ninety percent of the "performance" you pay for in Snowflake is actually just a tax on your inability to move your data. If you’re like most enginee…
Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row
Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double th…
One Histogram, Four Engines: Fixing the Statistics Silo Problem
TL;DR — DuckDB, DataFusion, Polars, and Postgres each compute and store table statistics their own way, so a histogram built in your ELT pipeline is…
Why Athena/Iceberg Tends to Make Code the Spec
Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole…
AI Enrichment Pipeline: From Sample Classification to 100K-File Metadata Search with Bedrock and OpenSearch NextGen
Quick Recap: What We Built in Part 1 In Part 1 , we built a metadata catalog on Apache Iceberg (S3 Tables) that makes unstructured files on FSx for…
From Hours to Seconds: An AI-Powered Metadata Catalog for Unstructured Data on FSx for ONTAP
What Works Now vs What Requires Validation This article separates verified AWS-native capabilities from cross-platform paths that still require vali…
Load PostgreSQL into Apache Iceberg with Sling
Introduction Apache Iceberg is the table format that turns a pile of Parquet files in object storage into something that behaves like a warehouse ta…
When does Iceberg beat Parquet+projection on AWS Glue, and when doesn't ?
Why this project I built this repo because I didn't have one of this kind yet and, having worked on data ingestion with Glue for a while, I wanted t…
Ursa — a new Diskless Lakestream engine for Kafka
The future of Kafka is diskless topics + native Apache Iceberg / Delta Lake integration Intro Ursa is a relatively new engine that's being fitted in…