Hosting an Entire Open Lakehouse on Port 8080
Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost ne…
AI & ML
⚑ Report a ProblemLatest AI & ML news from Tech News
All About Iceberg's File Format API: simplifying the engine/format interface and Parquet's multimodal future
What is the file format API? Apache Iceberg traditionally exposed its data files through three libraries, one each for Parquet, Avro and ORC, and un…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
What One Rust Client Can Reach Across Seven Iceberg Catalogs
This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works.…
Production RAG on the Lakehouse with BigQuery Vector Search and Apache Iceberg
As teams rush to build with Generative AI, they're creating a dangerous chasm between their data and AI stacks. This common architectural flaw intro…
Iceberg и Paimon — строительные блоки Streaming Lakehouse
Привет, Хабр! Я Алексей Новаков, ведущий инженер данных в Рунити. Сегодня разберемся, зачем Lakehouse понадобился streaming, почему одного Iceberg д…
Four Iceberg Tools, Three Agent Frameworks: What Ports, and What Doesn't
This article is a step by step build of one small Apache Iceberg data agent, written three times: in Google ADK, in AWS Strands and in Microsoft Age…
Four Iceberg Tools, Three Agent Frameworks: What Ports, and What Doesn't
This article is a step by step build of one small Apache Iceberg data agent, written three times: in Google ADK, in AWS Strands and in Microsoft Age…
Using Databricks FILE type against data on NAS: what works today with FSx for ONTAP S3 Access Points
A follow-up to Part 2: Databricks and FSx for ONTAP S3 Access Points in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some resu…
The query succeeded. Which table state did it read?
Why two successful queries can count different rows when they resolve different Delta or Iceberg table states. Two queries return different counts f…
The pipeline passed. The data still needs proof.
A pipeline job finishes without an error. The new table has 18% fewer rows. Should you rerun the pipeline, stop the release, or trust the new output…
Оптимизация MPP-кластера: предсказываем потребление памяти SQL-запросов
В аналитике больших данных системы массивных параллельных вычислений часто находятся под постоянной нагрузкой в режиме 24/7. Из десятков и сотен тыс…
Пока все хоронили пайплайны, ClickHouse достраивал слои
«Отдельные базы больше не нужны», «конец пайплайнов» - каждую неделю кто-то крупный со сцены хоронит то, что ты вчера поставил в прод. ClickHouse по…
[Databricks on AWS #0] The Target Architecture: Isolating Prod, Dev, and Sandbox with Unity Catalog
📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Ro…
Apache Paimon: steamhouse как логическое продолжение современных КХД
Apache Paimon: стриминговый lakehouse для дата-инженеров Сколько систем вы держите ради того, чтобы аналитики видели события через секунды, а годовы…
Using Microsoft Fabric Shortcuts to Avoid Duplicate Data Copies
Enterprise data platforms are really good at one thing: creating copies of the same data everywhere. Different teams copy the same curated folders i…
Только Сигма выбирают Delta Lake
Привет, Хабр! Меня зовут Дмитрий Кравчук, я занимаюсь всем, что связано с данными в блоке AI&ML MAGNIT TECH. Расскажу про фундамент прибыльных п…
Строим машину времени для данных (SCD-2) на движке Trino под управлением Airflow
Сегодня SCD-2-таблицы не только остаются актуальными для медленно меняющихся данных, но и, на мой взгляд, становятся гораздо проще в реализации благ…
DatAasee 0.9 Released
DatAasee Tech Stack Canvas Setting: Many distributed data and metadata sources Goals: Centralize metadata Interlinked metadata catalog Super-index f…