The Evaluation Debt You Don't Know You Have: Why Agent Evals Fail in Production
The Evaluation Debt You Don't Know You Have: Why Agent Evals Fail in Production By Paul Twist, Berlin | July 13, 2026 The Problem Nobody Talks About Y…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
tutorial
typescript
webdev
Workflow Series (05): Evaluation Framework — Three-Layer Testing and Trace Tracking
Why Workflows Need a Dedicated Evaluation Framework Traditional software testing covers code correctness. Workflows add two layers of uncertainty: LLM…
Reliable, and still wrong
A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but…
Your Model Upgrade Broke Three Workflows and the Tests Still Passed
Every team that runs agent evals eventually hits the same wall: your suite was green on Friday, you bumped the model from gpt-4.1 to gpt-4.1-2026-05 ,…
Evaluating Large Language Models: The Overfitting Problem
Introduction to Overfitting in LLM Evaluation We've all been there: you train a model, it performs exceptionally well on your test set, but when you d…
Who Grades the Grader? Your LLM Judge Is an Unvalidated Model in Production
Everybody's eval stack has the same load-bearing assumption nobody audits: that the model-as-judge is telling the truth. You wrote deterministic check…
Patronus AI lands $50M to build ‘digital worlds’ that stress-test AI agents
Agent-testing startup Patronus AI, founded by former Meta AI researchers, is experiencing nearly insatiable demand, its investor says.
Your RAG faithfulness check is measuring copy-paste, not faithfulness
I was building an eval harness for a retrieval-augmented generation pipeline, and the first faithfulness check I wrote was quietly wrong. It looked re…
Agent = Model x Harness: Your Eval Layer Is Part of the Agent, Not a Tool Beside It
There's a formula I keep coming back to when people ask why their slick demo agent falls apart in production: Agent = Model × Harness The model is the…
Your Agent Didn't Break, It Drifted: Detecting Slow Decay in Autonomous Systems
There is a specific kind of incident that no alert ever fires for, and it is the one I trust least. Nothing crashed. No exception, no 500, no failed h…
Stop Asking 'Is GAI Here' — Ask 'At What Layer'
Stop Asking 'Is GAI Here' — Ask 'At What Layer' The GAI debate has a structural problem. Someone says "passing this benchmark means GAI." A model pass…
Put Your Agent Evals in CI or Stop Calling Them Evals
Most teams I talk to have "evals." I ask them where the evals run. The answer is almost always the same: a notebook, a dashboard, a spreadsheet someon…
Evals Are Alignment Enforcement: Why Your Safety Strategy Needs Runtime Checks
The Argument The AI safety conversation is dominated by two camps: the alignment researchers thinking about existential risk, and the product engineer…
What is an LLM evaluation harness? A deep dive into lm-eval-harness
What is an LLM evaluation harness? A deep dive into lm-eval-harness You fine-tuned a 7B model. It aced your smoke tests, your colleague ran a few prom…
why Cohen's kappa drifts week to week (and what to do about it)
If your LLM-as-judge calibration kappa moves around week to week and you cannot explain it from labeller behavior, the usual cause is the marginal dis…