Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns
Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation Replacing a legacy ETL platform is…
Architecture
⚑ Report a ProblemLatest Architecture news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
news
opensource
performance
productivity
programming
python
react
security
showdev
softwareengineering
systemdesign
tutorial
typescript
webdev
Announcing det: - Fast, local-first SQL ETL workspace and pipeline orchestration engine
What if you could build complete, production-grade ETL pipelines in Pure SQL… without Jinja templates, without a database server, and without heavy…
Azure Synapse Databricks: A Migration Playbook That Survives Production
Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL…
Understanding Pega's `pzPVStream`: Why a 20-Year-Old Design Still Makes Sense
One of the most misunderstood parts of the Pega Platform is pzPVStream . Newcomers often describe it as a "black box", while experienced architects…
AgTech без хайпа: как мы строим систему, которая доводит агронома от симптома на поле до решения
Автор: Олег Линьков, Webformula ( webformula-agro.ru ) Агросектор — одна из самых недооценённых ниш для прикладного AgTech. Здесь нет хайпа вокруг L…
Как мы ушли от ETL к CDC: выбираем архитектуру real-time аналитики на PostgreSQL, Kafka и ClickHouse. Часть 1
Все началось с просьбы сделать отчеты в реальном времени. На первый взгляд задача выглядела простой, но довольно быстро выяснилось, что существующая…
Building My First End-to-End ETL Pipeline with Airflow, BigQuery, and Docker
Recently, I completed my first full Data Engineering project: building an end-to-end ETL pipeline using real-world Australian weather data spanning…
Why Metadata-Driven ETL Frameworks Scale Better Than Hardcoded Pipelines — and Where They Don't
Over the years, I've seen many data platforms start with good intentions. A few scripts are created to move data from one system to another, and eve…
A torrent search engine,for actually active torrents!
The core paradox of modern BitTorrent: how do you search without a central index? Searching needs metadata, an address that you can query. BitTorren…
Extract data from Databases into DuckLake
Introduction In the evolving landscape of data engineering, DuckLake is emerging as a powerful solution for building data lakes with ACID transactio…
ETL Pipeline: Fetching Real-Time News Data with Python and Postgres
The best way to actually understand data engineering is to build something that breaks, fix it, and watch it successfully run. In this article, we b…
Apache SeaTunnel Isn’t a Simple ETL Tool , Understanding Its DataFlow-Driven DAG Engine
In the field of data integration and synchronization, Apache SeaTunnel is undoubtedly one of the hottest tools today. This series will dive deep int…