Privatise your Data Streams with Bring Your Own Cloud (BYOC)
TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this mod…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
tutorial
typescript
webdev
File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables
For years, the open lakehouse had an honest gap that practitioners whispered about and slide decks skipped: encryption. Not the checkbox kind, every c…
Verifying How IAM and Lake Formation Behave for the Glue REST Catalog and S3 Tables
Original Japanese article : Glue REST CatalogとS3TablesのIAMとLake Formationの挙動を確かめる Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). In m…
GitOps for Geospatial Data: Building a Self-Healing, Zero-Cost Data Pipeline with GitHub Actions
Most data engineering and geospatial projects follow a predictable infrastructure blueprint: an ingestion cron job, an enterprise database (like PostG…
Scraping Indian government open data in 2026: what actually works
I maintain Village Finder , an open-source mapping project tracking over 78,000 Indian villages. It works by pulling daily raw updates straight from o…
Choose a Columnar Format From the Read Path Backward
The columnar-format ecosystem is moving quickly. Feature tables encourage teams to ask which format has the newest encoding. A system-design decision …
Feels Great To be part of this project, its a awesome experince to build this project and completed it on time
We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow…
wide CSV 여러 개를 EAV로 모아 gold mart 만들기
wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, …
schema drift를 fail이 아니라 warn으로 둔 이유
schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지…
source_hash로 같은 입력 재처리를 안전하게 skip하기
source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영…
Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take
Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The senten…
Epic Clarity Data Model Guide for Healthcare Data Engineers
Epic is installed at approximately 38% of US hospitals . If you work in healthcare data engineering long enough, you will encounter Epic Clarity. This…
Quantified Self 2.0: Stop Guessing Your Health History—Build a Personal Medical Vector Database
Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Me…
When LangGraph Succeeds but Silently Goes Wrong
The most dangerous failure is a clean run The most common mistake teams make when moving a LangGraph pipeline to production is assuming that a success…
Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables
Original Japanese article : Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する Introduction I'm Aki, an AWS Community Builder ( @jitepengi…
Designing an API-First Value-Based Care Analytics Stack for MA Payers
If you build software for Medicare Advantage (MA) plans, "analytics" usually arrives as a vague requirement and leaves as a pile of nightly batch jobs…
How do I answer "what did my data look like last month" in Postgres?
Sooner or later, someone asks: "What did this customer's plan look like when they signed up last February?" The current row won't tell you that; it's …
Your Data Pipeline Is Probably More Fragile Than You Think
Most engineering teams don't think much about their data pipelines until something breaks. That's partly because a healthy pipeline is almost invisibl…
OLTP vs OLAP : Understanding the Foundation of Modern Data Systems
Introduction Every business relies on data, but not all data workloads are the same. Processing a customer's payment, booking a flight, or updating in…
Debezium vs Managed CDC: How to Actually Decide Between Build and Buy
Most "Debezium vs managed tool" articles get the question wrong. They frame it as a product bake-off, feature grid included, and declare a winner. But…
Stop Digging Through PDFs: Build a FHIR-Standard EHR Knowledge Base with RAG
We’ve all been there: staring at a stack of printed lab results or a folder full of cryptic report_final_v2_NEW.pdf files, trying to remember if our c…
PowerBi is smart but also lazy
Introduction For the past few days I have been doing data cleaning using PowerBi 34 columns to be precise.Data cleaning and transformation to me alway…
What does electricity have to do with knowledge graphs?
Christopher Nolan? Crime? Not really... But this is what my first tries for a knowledge graph did. It wasn't hallucinating. There was a real directed_…
Query one Apache Iceberg table from Trino, Spark, and DuckDB
Query one Apache Iceberg table from Trino, Spark, and DuckDB Iceberg is useful because it lets multiple engines read the same table without copying da…
Day 57: Internals of ClickHouse® Data Parts and Merges – A Complete Guide
Introduction One of the reasons ClickHouse® can handle massive analytical workloads with exceptional speed is its storage architecture. Unlike traditi…
Data Engineer LeetCode: What to Grind and What to Skip
I spent about 80 hours grinding LeetCode before my first FAANG data engineering loop. Binary trees, dynamic programming, graph traversal. I could reve…
Day 53: ClickHouse® Data Sampling Explained – Query Billions of Rows Faster
ClickHouse® Data Sampling: Querying Billions of Rows Fast Day 53 of #100DaysOfClickHouse Modern analytics platforms routinely manage billions—or even …
# Turning Data Quality Checks into a Measurable Trust Score
Most data quality problems do not announce themselves loudly. They do not always break a pipeline. They do not always throw an exception. They do not …
Phase 1: Document Ingestion - The Hidden Complexity Before Embeddings
The Story Begins: Why Your Upload Button Is Just The Beginning 👦 Nephew: Uncle! I finally built my RAG system. User uploads a PDF, system finds answer…
Designing Enterprise Data Architecture: Lessons Beyond ETL
When people hear "data architecture," they often think about databases, ETL pipelines, or reporting tools. While those are important pieces, enterpris…