The Ultimate Quantified Self: Building a Private Health Knowledge Base with RAG (PKM for Health)
We've all been there: staring at a blood test report from three years ago, trying to remember if that "slightly elevated" glucose level was a one-time…
Architecture
⚑ Report a ProblemLatest Architecture news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
softwareengineering
systemdesign
tutorial
typescript
webdev
Entry-Level Data Engineering Is Gone. Here's the Proof.
I've been on both sides of the data engineering hiring table for years. Interviewed candidates, been the candidate, watched the market shift underneat…
AI Data Lake and Lakehouse
The word "AI" appears in every data platform pitch now. Most of the time, it means nothing — a chatbot bolted onto a dashboard, an LLM that generates …
One Copy, Two Engines: Architecting a Governed Lakehouse Across Fabric and Azure Databricks
For most of the last decade, the honest answer to "should we standardize on Databricks or on the Microsoft data platform?" was: pick one and live with…
🔐 Storage Credentials in Databricks Unity Catalog (AWS) — The Security Backbone You Can't Ignore
As organisations move to the Databricks Lakehouse Platform on AWS, two priorities consistently surface at the top of every architecture conversation: …
Kafka Topics, Partitions, and Offsets Explained: The Concepts Every Beginner Confuses
When I first started learning Apache Kafka, I quickly realized that three terms appeared everywhere: topics, partitions, and offsets. At first glance,…
Why Athena/Iceberg Tends to Make Code the Spec
Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole sy…
Why AI Projects Fail Even with Great Models
Artificial Intelligence is advancing at an incredible pace. Every week, we see announcements about new Large Language Models (LLMs), improved reasonin…
What Is a Semantic Layer? A Practical Guide for Data Engineers
Your data warehouse has a table called orders . It has columns like amount , status , created_at , and customer_id . Now three people ask "What was Q1…
Real-Time Analytics: When You Need It and When You Don't
"We need real-time analytics" is one of the most common requests in data engineering. It's also one of the most misunderstood. When the VP of Sales sa…
The AI Cheat Tool Your Interview Cannot See
I've been on hiring panels where we spent 45 minutes convinced a candidate was sharp. Articulate answers. Clean code. Solid reasoning. Then in the deb…
Exactly-Once Semantics in Kafka: Promise vs. Reality
"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and…
I moved 10 million rows in 9.9 seconds with pip install apitap
Last week I published apitap - an open-source engine that moves whole tables between databases. One function, no config files, no pipeline DAGs: impor…
Privatise your Data Streams with Bring Your Own Cloud (BYOC)
TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this mod…
File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables
For years, the open lakehouse had an honest gap that practitioners whispered about and slide decks skipped: encryption. Not the checkbox kind, every c…
Verifying How IAM and Lake Formation Behave for the Glue REST Catalog and S3 Tables
Original Japanese article : Glue REST CatalogとS3TablesのIAMとLake Formationの挙動を確かめる Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). In m…
GitOps for Geospatial Data: Building a Self-Healing, Zero-Cost Data Pipeline with GitHub Actions
Most data engineering and geospatial projects follow a predictable infrastructure blueprint: an ingestion cron job, an enterprise database (like PostG…
Scraping Indian government open data in 2026: what actually works
I maintain Village Finder , an open-source mapping project tracking over 78,000 Indian villages. It works by pulling daily raw updates straight from o…
Choose a Columnar Format From the Read Path Backward
The columnar-format ecosystem is moving quickly. Feature tables encourage teams to ask which format has the newest encoding. A system-design decision …
Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take
Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The senten…
Epic Clarity Data Model Guide for Healthcare Data Engineers
Epic is installed at approximately 38% of US hospitals . If you work in healthcare data engineering long enough, you will encounter Epic Clarity. This…
Quantified Self 2.0: Stop Guessing Your Health History—Build a Personal Medical Vector Database
Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Me…
When LangGraph Succeeds but Silently Goes Wrong
The most dangerous failure is a clean run The most common mistake teams make when moving a LangGraph pipeline to production is assuming that a success…
How do I answer "what did my data look like last month" in Postgres?
Sooner or later, someone asks: "What did this customer's plan look like when they signed up last February?" The current row won't tell you that; it's …
Your Data Pipeline Is Probably More Fragile Than You Think
Most engineering teams don't think much about their data pipelines until something breaks. That's partly because a healthy pipeline is almost invisibl…
OLTP vs OLAP : Understanding the Foundation of Modern Data Systems
Introduction Every business relies on data, but not all data workloads are the same. Processing a customer's payment, booking a flight, or updating in…
Debezium vs Managed CDC: How to Actually Decide Between Build and Buy
Most "Debezium vs managed tool" articles get the question wrong. They frame it as a product bake-off, feature grid included, and declare a winner. But…
Stop Digging Through PDFs: Build a FHIR-Standard EHR Knowledge Base with RAG
We’ve all been there: staring at a stack of printed lab results or a folder full of cryptic report_final_v2_NEW.pdf files, trying to remember if our c…
Query one Apache Iceberg table from Trino, Spark, and DuckDB
Query one Apache Iceberg table from Trino, Spark, and DuckDB Iceberg is useful because it lets multiple engines read the same table without copying da…
Day 57: Internals of ClickHouse® Data Parts and Merges – A Complete Guide
Introduction One of the reasons ClickHouse® can handle massive analytical workloads with exceptional speed is its storage architecture. Unlike traditi…