Pick the LLM eval metric before you build the dashboard
You can ship a polished eval dashboard and still learn nothing about whether the product works. A common way to get there is picking a number becaus…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
RAG Evaluation & Monitoring: Metrics, Frameworks & Production Observability
RAG Evaluation & Monitoring: Metrics, Frameworks & Production Observability Comprehensive guide to evaluating and monitoring Retrieval-Augme…
I rejected a model that passed everyone else's benchmark
I vendored an audio deepfake detector into my verification platform. It loaded cleanly, every integration check passed, and the forward pass looked…
7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)
You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on.…
AI Model Evaluation: Best Practices for Testing and Validation
AI Model Evaluation: Best Practices for Testing and Validation Evaluating AI models is critical for ensuring quality, safety, and reliability. In th…
The benchmark that disproved its own result
An article came past claiming a two-year-old coding-trained model beats a newer general model at editing structured text that isn't code. Its thesis…
Обновление бенчмарка MERA Text 2.0
Всем привет, с вами команда MERA. В прошлом анонсе мы рассказали о новых публичных тестах на рассуждения и обещали, что следующим большим обновление…
My board never scored an outage as a regression. My evidence couldn't prove it.
I put a preprint on arXiv this week. It is about forging evidence bundles that my own verifier calls clean. I mutated the sites where the verifier i…
RAG без магии: что действительно влияет на качество системы
У большинства разработчиков, не работающих с ИИ, ментальная модель RAG либо ошибочна, либо опасно неполна. И дело не в том, что они плохие инженеры,…
Writing the code is no longer the bottleneck
The industry's currently obsessed with how fast we can generate code. Every morning in our engineering general channel on Microsoft Teams, someone's…
We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.
The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41%…
Choosing the Right LLM-as-a-Judge: A Practical Guide with Model Recommendations
Choosing the Right LLM-as-a-Judge: A Practical Guide with Model Recommendations If you're building AI systems—whether RAG bots, generative models, o…
Your Golden Dataset Is Rotting: The Eval Oracle Nobody Re-Validates
We talk about agents drifting. We almost never talk about the thing we measure them against drifting. But your golden dataset — the fixtures, expect…
GuardRate: как мы построили независимую арену для guardrail-моделей (часть 1)
Всем привет, на связи команда HiveTrace! Мы уделяем много времени разработке собственных моделей и часто задаемся вопросом: "какой из двух гар…
I Built an Agent Evaluation Harness for Local AI — What Most People Get Wrong
I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes…
How EvalPort's Grader System Works: 11 Types for LLM Evaluation
How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own wa…
Your New Eval Rule Is Untested Code Guarding Production
You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate. Congratulations — you now have a piece of untested…
OpenEval: Why LLM Evaluation Needs a Standard Format
Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo,…
An LLM judge is a biased instrument, not a measurement
Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Vari…
Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation
Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with type…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92%…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92%…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92%…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92%…
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%
Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured,…