Our integration tests skipped themselves on every Dependabot pull request
In April our checkout started failing for every customer who paid by bank transfer. The cause was a minor version of our payment provider's SDK, mer…
Open Source
⚑ Report a ProblemLatest Open Source news from Tech News
All topics
AI
agents
ai
api
architecture
automation
backend
beginners
career
database
devchallenge
devops
discuss
javascript
linux
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Our release workflow tagged three versions that were never published
In March a customer opened an issue on our public SDK repository: the changelog announced version 3.8.0, the GitHub release page had notes for it, a…
A refactor that touched no database changed our database password
In May I merged a pull request that moved our tagging into a shared locals block. Twelve files, no resources added or removed, a plan reviewed by tw…
Our migration test passed because the last job left its database behind
In July a database migration passed every check in our pipeline and then failed in staging on its first statement. It created an index using a funct…
One message we could not parse stopped four hours of dispatches
At twelve minutes past four on a Saturday morning, one partition of our shipments topic stopped moving. We found out at nine, when a depot rang to a…
Every question we asked our logs took six minutes to answer
During a fifty minute incident in June, four of us sat in a call watching a spinner. Every search we ran against our log store took between six and…
Our rate limit punished everyone except the customer causing the problem
At twenty past two on a Tuesday morning one customer began a historical backfill against our API. Within ten minutes a hundred and twelve other cust…
One new label multiplied our metrics by every order we take
On a Friday afternoon a colleague added a label to a counter so he could follow one customer's orders through a queue. The label was the order id. I…
The configuration we changed in June arrived one pod at a time in August
A support ticket said our dispatch emails were going out twice. Not always, not to any customer we could pick out, and never while we watched. Twent…
The timeline we built from the logs was physically impossible
We spent two hours of a payment incident arguing about a duplicate request that never existed. The timeline we had assembled from the logs showed th…
A third of our releases went out through the emergency path
A deploy at eleven on a Tuesday night broke search for about forty minutes. It had skipped the integration suite and the staging soak, and it had ne…
One instance was bad at its job and nothing was designed to notice
Checkout errors sat between three and eight percent for most of a Thursday. Not enough to page, because our threshold is one percent sustained acros…
The version number that promised nothing had changed
Six services failed to start within about forty minutes of each other on a Wednesday morning, all with the same error about a missing configuration…
The shell step that could not fail no matter what we did to it
Our integration suite had been green for eleven weeks. That is not a boast, it is the symptom. A colleague onboarding to the repo wrote a deliberate…
The hotfix somebody made by hand that our nightly apply reverted
We had an incident at 19:20, fixed it at 19:35, wrote it up, and went home. The same incident started again at 02:07 the next morning with nobody aw…
The HTTP client with no timeout that took down four services
A recommendations service, which is genuinely optional, started responding slowly. Not failing. Responding, in ninety seconds instead of eighty mill…
The migration that shipped with the code and locked the table for eleven minutes
The release added one nullable column and an index to the orders table. It had been reviewed, it ran in under a second in staging, and it was bundle…