Engineering Resilient Systems for a Magnitude 5.7 Earthquake in Pakistan
Observed 2026‑10‑17 – USGS event page The sudden release of strain that generated a 5.7‑magnitude event in Pakistan highlights the need for engineer…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
Deduplicated Fallback Query Uses Wrong Relevance Threshold
Deduplicated fallback search queries can accidentally trigger the wrong relevance threshold and reject valid results when the requested query matche…
Exponential Backoff Isn't Enough: The Retry Storm Hiding in Your Recovery Path
The problem A dependency blips for 400ms — a deploy, a GC pause, a brief network partition, doesn't matter which. Every client with an in-flight req…
The Agent Rewrote the Shell Script in Python and Dropped the Exit Code
The build script was 40 lines of bash. It ran npm test , then npm run build , then rsync the dist folder to a server, then paged us on Slack. It wor…
Production failures: the things that page you · 3. Retry storms
Production failures: the things that page you · Chapter 3 of 24 · Reliability · new chapter every Friday morning By the end of this chapter: See how…
Kill the bug class, not just the bug
The dry run said the reminder text would go out to a customer with the window "12PM–2PM." The actual appointment was 8AM–10AM. Four hours off, on a…
Don't make the failure path stricter than the success path
Two incidents in one week, same anti-pattern. In both cases, the system didn't fail because of the failure — it failed because of how we handled the…
The sync that deleted our leads every 60 seconds
The sync that deleted our leads every 60 seconds Leads were vanishing. Not bouncing, not erroring — just gone. New leads would appear in our registr…
How many jobs should a Node.js queue run at once?
Part 2 of 10: Reliable background jobs in Node.js, from zero . By Sandro Dezerio , senior full-stack engineer. The signup scenario is made up, the s…
Honoring Margaret Hamilton: Preserving Legacy of Pioneers Behind Modern AI and Autonomous Systems
Technical Reconstruction of Margaret Hamilton's Contributions to Software Engineering and AI Foundations Main Thesis: Margaret Hamilton's pioneering…
We migrated the receiver but forgot the return trip
We migrated the receiver but forgot the return trip Same incident, different lesson. (If you're counting, this is the day our infrastructure decided…
Compare Startup SMS Alerts Provider Templates — 4 US-EU Signature Tests
Operational constraint: a startup that needs to compare an SMS alerts provider cannot treat every failed message as proof that a recipient is invali…
Tunnels are for dev: why our production webhooks left localhost.run
Tunnels are for dev: why our production webhooks left localhost.run This is the third post in a trilogy I didn't plan. Post #1 was the diagnosis: my…
I published the same two posts twice and didn't notice for a month
An audit script I wrote printed a line I had to read three times: ERROR live duplicate x2 (ids 4532492, 4532387): DeepSeek raised its prices in the…
Transactional Email Operations: Compliance, Custom Domains, and Bounce Reconciliation
A signup system has one unforgiving constraint: a verification link that arrives late, twice, or not at all is a reliability failure at the exact mo…
Approval timeouts: what your agent should do when nobody answers
§ 01 · The missing fifth field In the approval queue pattern I described four fields every approval item should carry: the action, the justification…
Preserve First: Recovery Design for Local Creative Data
The most dangerous button in a creative tool is not "delete." It is the recovery dialog that appears when something is already wrong: "We couldn't l…
One message we could not parse stopped four hours of dispatches
At twelve minutes past four on a Saturday morning, one partition of our shipments topic stopped moving. We found out at nine, when a depot rang to a…
Our rate limit punished everyone except the customer causing the problem
At twenty past two on a Tuesday morning one customer began a historical backfill against our API. Within ten minutes a hundred and twelve other cust…
DNS Record Write Rejected Because Zone ID Is Not Domain Name — Validation Debug
Short answer: when a DNS record write is rejected because a zone ID is not a domain name, resolve the opaque reference first, canonicalize the retur…
We scheduled a question for an hour when nobody could answer it
The run was fully unattended. That was the point. Everything upstream of the final step had been automated for weeks, and the last remaining human t…
What "Fully Automated" Actually Costs
I run an automated system on a remote machine with nobody watching it. When I started, I described it to people as passive. Over two days it needed…
One instance was bad at its job and nothing was designed to notice
Checkout errors sat between three and eight percent for most of a Thursday. Not enough to page, because our threshold is one percent sustained acros…
Feature flags are the loaded gun modern deploys pretend is safe
A trading firm experienced a loss of $440 million within 45 minutes. This was not due to a cyber attack or a sudden crash in the market. It was a fo…
The HTTP client with no timeout that took down four services
A recommendations service, which is genuinely optional, started responding slowly. Not failing. Responding, in ninety seconds instead of eighty mill…
Claude Keeps Going Down, and Anthropic's Own Status Page Says So
Open Claude on a weekday in August 2026 and there’s a fair chance you’ll meet a spinner, an error, or the words “server is busy” where an answer sho…
Asynchronous Form Schema Discovery: Idempotent Retries and Secure Temporary Files
Use a durable job record as the unit of work for form schema discovery: validate the document at admission, submit one asynchronous extraction job u…
Retries didn't make GitHub resilient. They DDoS'd it.
A capacity blip at GitHub resulted in an outage that lasted 7 hours and 47 minutes. The infrastructure recovered faster than the clients let it. 😬 W…
What Idempotent Actually Means: Why Retries Are Safe (and When They Aren't)
Originally published on Loop & Retry — field notes on building LLM agents that survive production. Search for "what does idempotent mean" and yo…
Your Backup Doesn't Exist Until You've Restored It
Everyone has backups. Almost nobody tests them. And an untested backup is not a backup — it's a hope, wearing a backup's uniform. I learned this the…