95% recall, 80% answers: what LongMemEval retrieval scores don't tell you
Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was o…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
Прогнали GigaChat 3.5 по реальным кейсам: Результаты большого тест‑драйва
GigaChat 3.5 вышел в июле этого года, а в начале сентября были выложены открытые веса этой модели. И многие сейчас задаются…
Приниматоры решений: бенчмарк на русском для Jev-style моделей
Последние пару лет мы довольно странно используем большие языковые модели. Из-за их умения делать всё мы пытаемся решить с их помощью всё — роутинг…
Half the MCP servers that answer you don't actually work
A dated, reproducible health probe of 78 registry-listed MCP servers. Pennyforge Studio · 2026-10-05 · second dated point 2026-10-06 · repro + raw d…
Claude Opus 5.5: 40% cheaper, frontier-grade performance
This week was dominated by cost compression and gateway consolidation. Claude Opus 5.5 dropped with a meaningful price cut and no code changes requi…
Последний пеликан
Что-то много пеликанов на велосипеде начало появляться в ленте моего информационного пузыря в твиттер и я решил провести собственный эксперимент: GP…
I Tested Q4_K_M vs MXFP4 on the Same Laptop — The Supposedly-Faster New Format Lost
I tested two local LLMs in two different quantization formats on the same laptop, on the same prompt, three trials each. The result is the opposite…
I Tested 4 LLM APIs for Coding Work — All 20 Runs Passed, But Speed Varies 10x
I tested 3 free LLM API providers and one paid Claude plan on the same 5 real coding tasks. All 20 runs passed on first attempt. The story isn't whe…
Tried `sponsors/wonderwhy-er`: A Practical MCP Server Review
Tried sponsors/wonderwhy-er : A Practical MCP Server Review sponsors/wonderwhy-er is an MCP server that gives Claude a controlled interface to devel…
We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model.
We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row. The LangChain…
A LongMemEval-S number you can reproduce
We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no w…
I Tested GLM-5.3-Flash and Qwen3.8-Flash on 24 Real Tasks
I test-ran both of this week's open-weight flash models against 24 small, real workloads from an actual product stack — structured extraction, SEO m…
We tested "tokenize before you compress" against 452 configurations, and it mostly held up
A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text…
Qwen3.8-27B на своей карте против 304B по API
Закончил тесты и подготовил результаты этих тестов по самой новой, последней, крутой (и способной влезть на карту, что можно купить домой - RTX5090,…
What should an MCP tool return? I ran 72 trials instead of arguing
There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, fo…
Benchmarking AI Coding Agents on Real Pull Requests
No synthetic puzzles, no contaminated suites: 25 tasks harvested from PRs merged in 2026 across five languages, graded by each project's own held-ou…
AI Daily Digest — August 1, 2026: ARC-AGI-3 Harness Discovery, EU AI Gigafactories, Devin SWE-1.7
🤖💻 AI Daily Digest — August 1, 2026 OpenAI Shows How Two Harness Settings Tripled ARC-AGI-3 Scores OpenAI published a rare technical deep-dive on Ju…
.NET Matrix: взвешенный выбор библиотеки, а не по звёздам на GitHub
.NET Matrix — это новый открытый проект, который сравнивает .NET-библиотеки внутри одной категории по трём аспектам: возможности , скорость и исполь…
39,000 Torrents: The Bug My Green Benchmark Never Caught
[Torrent911] Inception.2010.TRUEFRENCH.1080p.BluRay.x264-YIFY That's a real filename. The goal: extract "Inception" and "2010" from it, query the TM…
OpenRouter vs Vercel vs LLMGateway Performance
Every AI gateway adds a hop between your app and the model. The question that matters is what that hop costs at the moment your user is staring at a…
Model Showdown Round 9: Qwen 3.6 27B vs Qwen 3.6 35B-A3B vs Qwythos-9B vs GLM-4.7-Flash vs Nemotron-3-Nano
Round 7 ended on a cliffhanger I couldn't stop thinking about. Qwen 3.6 35B-A3B built the entire feature — read the codebase, wrote the files, got a…
AdvancedMathBench: A New Benchmark for LLM Advanced Mathematical Reasoning
What Changed Large language models (LLMs) have demonstrated proficiency in high-school and olympiad-style mathematics. However, their performance in…
AI Coding Tools Benchmark 2026: Cursor vs Copilot vs Windsurf vs Claude Code
I spent two weeks testing Cursor, GitHub Copilot, Windsurf, and Claude Code on the same set of tasks. Not vibes. Not feature lists. Actual work: bui…
GLM Is the New Hotness, So Let's Test It On the Homelab
GLM is the new hotness. I'm hearing it from both sides of the AI builder world. Software engineers are talking about it because the benchmark number…
Harness Bench: как оценить агентский harness и выбрать связку с моделью
Привет! Я Андрей Иванов, NLP-исследователь в R&D-лаборатории red_mad_robot. Когда мы собираем AI-агента, первым делом выбираем модель под задачу…
Too cheap to be good? Think again.
For years, I ran my WordPress sites on OpenLiteSpeed. Fast server, LSCache is genuinely impressive, and the OLS/WordPress combo is hard to beat on r…
A UMAP With Arrows Is Not a Benchmark. This Is
How I built a three-task evaluation framework for RNA velocity trajectory inference -- measuring global ordering, pairwise rank preservation, and ro…
Engineering CellFateBench: A Reproducible Python Benchmark for Single-Cell Genomics Reasoning
CellFateBench is a scientific software and benchmark-engineering project for evaluating reasoning over single-cell genomics workflows. The project w…
Иллюзия 99% F1 в Time Series: как искажаются метрики в детекции аномалий и что показывает реальный тест 14 архитектур
Многие свежие SOTA-статьи по детекции аномалий во временных рядах заявляют F1 ≈ 99%. Мы проверили один из таких методов, и оказалось, что волшебство…
Локальный запуск openai/gpt-oss-20b MXFP4 GGUF на ноутбуке без дискретной видеокарты: практический тест на 32 GB RAM
Запустил openai/gpt-oss-20b в варианте MXFP4 GGUF на обычном ноутбуке без дискретной видеокарты: только CPU, встроенная Radeon 780M и общая оператив…