Pick the LLM eval metric before you build the dashboard
You can ship a polished eval dashboard and still learn nothing about whether the product works. A common way to get there is picking a number becaus…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
- игры
AI
News
Tech
agents
ai
api
architecture
automation
backend
beginners
career
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Период полураспада бенчмарка ИИ сократился вдвое: 19 месяцев вместо 38
В мае я подал OenoBench, винный бенчмарк для языковых моделей, на NeurIPS. В дискуссии с ревьюерами всплыл вопрос, на который ни у меня, ни у них не…
RAG Evaluation & Monitoring: Metrics, Frameworks & Production Observability
RAG Evaluation & Monitoring: Metrics, Frameworks & Production Observability Comprehensive guide to evaluating and monitoring Retrieval-Augme…
I rejected a model that passed everyone else's benchmark
I vendored an audio deepfake detector into my verification platform. It loaded cleanly, every integration check passed, and the forward pass looked…
Jev vs Local Models: Where Japanese Intent Routing Stands After 1,000 Utterances
📝 Originally published (in Japanese) at forge.workstyle.tech . The router that decides "which process to assign this phrase to" is crucial for the u…
Как я собрал винный бенчмарк на 3 266 вопросов руками LLM и где они меня подвели
Полгода назад я решил измерить, сколько языковые модели на самом деле знают про вино. Вышел бенчмарк OenoBench: 38 104 факта из открытых источников,…
7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)
You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on.…
How LLM Evaluation Actually Works: Inside the Satellite Geo QCM Leaderboard
How LLM Evaluation Actually Works: Inside the Satellite Geo QCM Leaderboard TL;DR: LLM evaluation only produces comparable numbers when every model…
AI Model Evaluation: Best Practices for Testing and Validation
AI Model Evaluation: Best Practices for Testing and Validation Evaluating AI models is critical for ensuring quality, safety, and reliability. In th…
They knew it wasn't the model. They patched it anyway.
Two articles ago I described a habit: when a system has a language model inside it and the output wobbles, the explanation drifts to the model. Then…
GuardRate: метрики, бенчмарки и инсайты из первого аудита (часть 2)
В первой части мы верхнеуровнево рассмотрели результаты оценки 37 моделей, обсудили проблему оценки и сравнения guardrail моделей, рассказали, как л…
GuardRate: метрики, бенчмарки и инсайты из первого аудита (часть 2)
В первой части мы верхнеуровнево рассмотрели результаты оценки 37 моделей, обсудили проблему оценки и сравнения guardrail моделей, рассказали, как л…
The benchmark that disproved its own result
An article came past claiming a two-year-old coding-trained model beats a newer general model at editing structured text that isn't code. Its thesis…
How to evaluate a RAG system: recall, faithfulness and the questions that matter
You cannot improve a RAG system you are not measuring, and "I asked it five questions and it seemed fine" is not measuring. Every serious change I m…
Обновление бенчмарка MERA Text 2.0
Всем привет, с вами команда MERA. В прошлом анонсе мы рассказали о новых публичных тестах на рассуждения и обещали, что следующим большим обновление…
My board never scored an outage as a regression. My evidence couldn't prove it.
I put a preprint on arXiv this week. It is about forging evidence bundles that my own verifier calls clean. I mutated the sites where the verifier i…
RAG без магии: что действительно влияет на качество системы
У большинства разработчиков, не работающих с ИИ, ментальная модель RAG либо ошибочна, либо опасно неполна. И дело не в том, что они плохие инженеры,…
Writing the code is no longer the bottleneck
The industry's currently obsessed with how fast we can generate code. Every morning in our engineering general channel on Microsoft Teams, someone's…
We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.
The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41%…
Choosing the Right LLM-as-a-Judge: A Practical Guide with Model Recommendations
Choosing the Right LLM-as-a-Judge: A Practical Guide with Model Recommendations If you're building AI systems—whether RAG bots, generative models, o…
Your Golden Dataset Is Rotting: The Eval Oracle Nobody Re-Validates
We talk about agents drifting. We almost never talk about the thing we measure them against drifting. But your golden dataset — the fixtures, expect…
GuardRate: как мы построили независимую арену для guardrail-моделей (часть 1)
Всем привет, на связи команда HiveTrace! Мы уделяем много времени разработке собственных моделей и часто задаемся вопросом: "какой из двух гар…
I Built an Agent Evaluation Harness for Local AI — What Most People Get Wrong
I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes…
How EvalPort's Grader System Works: 11 Types for LLM Evaluation
How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own wa…
Your New Eval Rule Is Untested Code Guarding Production
You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate. Congratulations — you now have a piece of untested…
Your Agent's Deadline Is a Correctness Test, Not an SLO
Ask an engineer to list their agent's failure modes and you'll hear about hallucinations, wrong tool calls, and bad JSON. Ask about time and you get…
OpenEval: Why LLM Evaluation Needs a Standard Format
Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo,…
502 на ingress, 302 в приложении: как я учил инфраструктурного LLM-агента не врать
Я делаю внутреннего read-only LLM-агента для инфраструктурных расследований. Инженер задаёт вопрос обычным языком, а агент собирает доказательства и…
An LLM judge is a biased instrument, not a measurement
Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Vari…
The Cold-Start Problem for Agent Evals: What to Gate on Day One With Zero Labeled Data
You just shipped an agent. It works in the demo. Now someone asks the reasonable question: "How do we know it keeps working?" And you reach for eval…