Последние новости

A LongMemEval-S number you can reproduce

We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no w…

I Tested GLM-5.3-Flash and Qwen3.8-Flash on 24 Real Tasks

I test-ran both of this week's open-weight flash models against 24 small, real workloads from an actual product stack — structured extraction, SEO m…

We tested "tokenize before you compress" against 452 configurations, and it mostly held up

A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text…

What should an MCP tool return? I ran 72 trials instead of arguing

There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, fo…

Benchmarking AI Coding Agents on Real Pull Requests

No synthetic puzzles, no contaminated suites: 25 tasks harvested from PRs merged in 2026 across five languages, graded by each project's own held-ou…

AI Daily Digest — August 1, 2026: ARC-AGI-3 Harness Discovery, EU AI Gigafactories, Devin SWE-1.7

🤖💻 AI Daily Digest — August 1, 2026 OpenAI Shows How Two Harness Settings Tripled ARC-AGI-3 Scores OpenAI published a rare technical deep-dive on Ju…

39,000 Torrents: The Bug My Green Benchmark Never Caught

[Torrent911] Inception.2010.TRUEFRENCH.1080p.BluRay.x264-YIFY That's a real filename. The goal: extract "Inception" and "2010" from it, query the TM…

MCPMark v2: InsForge on Sonnet 4.6

Originally published on the InsForge blog , written by Tony Chang (CTO & Co-Founder). Reposted here with permission. In December we published th…

OpenRouter vs Vercel vs LLMGateway Performance

Every AI gateway adds a hop between your app and the model. The question that matters is what that hop costs at the moment your user is staring at a…

Can You Beat an LLM? Building Humans vs. Humanity's Last Exam

Frontier models are crushing benchmark after benchmark...so I built a quiz to ask this simple yet humbling question: can a human still beat them? Hu…

The Same RTX 5090, but the GPU Sat Idle — a CPU-Bound Go Solver and the Case for L2 Cache

This is the second in a short series that benchmarks a single RTX 5090 by re-running published Go solvers — programs that don't just play Go but pro…

One RTX 5090 vs a 12-GPU Cluster — Benchmarking a Decade of GPUs on the Same Go Proof

You don't need to know anything about Go to read this. The game is just the fixed yardstick. The story is a hardware benchmark: the same program, th…

Model Showdown Round 9: Qwen 3.6 27B vs Qwen 3.6 35B-A3B vs Qwythos-9B vs GLM-4.7-Flash vs Nemotron-3-Nano

Round 7 ended on a cliffhanger I couldn't stop thinking about. Qwen 3.6 35B-A3B built the entire feature — read the codebase, wrote the files, got a…

AdvancedMathBench: A New Benchmark for LLM Advanced Mathematical Reasoning

What Changed Large language models (LLMs) have demonstrated proficiency in high-school and olympiad-style mathematics. However, their performance in…

Which LLM should I actually code with? I built a small benchmark to find out

LLM code benchmark — Peculiar Engineer A small, self-run coding benchmark: 3 models on 14 problems across 3 languages, scored on pass@k, cost, and s…

I Benchmarked 42 Compression Formats Spanning Four Decades. Here's What to Actually Use.

I run ezyZip , a browser-based archive tool, so "which format should I use?" is a question I field constantly. The honest answer is usually "it depe…

AI Coding Tools Benchmark 2026: Cursor vs Copilot vs Windsurf vs Claude Code

I spent two weeks testing Cursor, GitHub Copilot, Windsurf, and Claude Code on the same set of tasks. Not vibes. Not feature lists. Actual work: bui…

I built a neutral benchmarking layer for quantum simulators in Rust — and it revealed a silent disagreement between two backends

GLM Is the New Hotness, So Let's Test It On the Homelab

GLM is the new hotness. I'm hearing it from both sides of the AI builder world. Software engineers are talking about it because the benchmark number…

Too cheap to be good? Think again.

For years, I ran my WordPress sites on OpenLiteSpeed. Fast server, LSCache is genuinely impressive, and the OLS/WordPress combo is hard to beat on r…

A UMAP With Arrows Is Not a Benchmark. This Is

How I built a three-task evaluation framework for RNA velocity trajectory inference -- measuring global ordering, pairwise rank preservation, and ro…

Engineering CellFateBench: A Reproducible Python Benchmark for Single-Cell Genomics Reasoning

CellFateBench is a scientific software and benchmark-engineering project for evaluating reasoning over single-cell genomics workflows. The project w…

LLM Wire Format Benchmark: Which Format Can AI Actually Read and Write?

Every LLM wire format claims token savings. Nobody proves whether AI models can actually comprehend the format at scale, or produce valid output in…

Ideogram 4.0 is Good. Just Good.

A blind test across 240 images and 10 professional designers just dropped. Ideogram 4.0 against Gemini 3.1, Grok Imagine, and FLUX.2 Max. The result…

How fast is LlamaStash? Overhead, throughput, and a fair comparison with Ollama and LM Studio

Originally published at deepu.tech . In my release post for LlamaStash I made a claim I need to back up. The wrapper adds zero overhead vs running l…

Benchmarking the Claude Agent SDK on a local LLM: Haiku and Sonnet tier performance

The Claude Agent SDK exposes three budget tiers ( haiku , sonnet , opus ) and reads its routing target from environment variables on every call. Tha…

I Benchmarked 17 ESLint Security Plugins. Only One Found Every Vulnerability.

Skip to: Full Results | Category Breakdown | The Leaderboard | Methodology TL;DR I built a benchmark suite with 40 vulnerable code patterns across 1…

Multi-Shot vs Zero-Shot: When Adding Examples Actually Hurts Accuracy

Book: Prompt Engineering Pocket Guide: Techniques for Getting the Most from LLMs Also by me: Thinking in Go (2-book series) — Complete Guide to Go P…

LMR-BENCH: Can LLM Agents Reproduce NLP Research Code? (EMNLP 2025)

A research team from the University of Texas at Dallas published LMR-BENCH at EMNLP 2025, asking a specific question: can LLM agents reproduce the c…

AI-generated accessibility, an update — frontier models still fail, but skills change the game

A few months ago I shared early results from the A11y LLM Eval project, a benchmark that measures how accessibly LLMs generate UI code. The previous…