95% recall, 80% answers: what LongMemEval retrieval scores don't tell you
Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was o…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
Our agentic coding benchmark results are now an open dataset on Hugging Face
I've put our benchmark numbers on Hugging Face, as an open dataset with a small leaderboard Space on top, so anyone can pull the rows and argue with…
Llmspeed Benchmark: Pick the Perfect On‑Device LLM Hardware Fast
Llmspeed Benchmark: Your Practical Guide to Choosing the Right Hardware for On‑Device LLMs Introduction If you’re tired of guessing whether a single…
Kandinsky 6.0 on a 16 GB GPU: the official recipe runs out of memory, and only the 29B model passed my Spanish test
Kandinsky 6.0 generates five seconds of video with its audio , voices included, from a text prompt (MIT license). I wanted to know two things: does…
디시전 AI 글로벌 벤치마크 'S1MB' 1위: 퍼플렉시티·클라우드플레어·메타 제친 Darwin-27B-ZTC-v5
요약(TL;DR): 비드래프트의 판정 엔진 Darwin-27B-ZTC-v5가 허깅페이스 공개 벤치마크 S1MB에서 152개 모델 중 종합 1위에 올랐다. 137개 벤치마크 순위를 점수로 바꿔 평균한 Borda 점수는 91.16으로, 2위 퍼플렉시티(88.66)를 2…
Half the MCP servers that answer you don't actually work
A dated, reproducible health probe of 78 registry-listed MCP servers. Pennyforge Studio · 2026-10-05 · second dated point 2026-10-06 · repro + raw d…
Claude Opus 5.5: 40% cheaper, frontier-grade performance
This week was dominated by cost compression and gateway consolidation. Claude Opus 5.5 dropped with a meaningful price cut and no code changes requi…
How LLM Evaluation Actually Works: Inside the Satellite Geo QCM Leaderboard
How LLM Evaluation Actually Works: Inside the Satellite Geo QCM Leaderboard TL;DR: LLM evaluation only produces comparable numbers when every model…
A code review benchmark that isn't the vendor ranking itself
Ask which AI code review tool is best and the answer you get depends on who is publishing it. The deepsource.com listicle ranks CodeRabbit first and…
JetBrains Ranked AI Agents on Real Kotlin Projects. The Token Column Is the Real Story.
JetBrains released the Kotlin Benchmark , its official benchmark for grading AI coding agents on real Kotlin engineering work. Claude Code with Opus…
Two "Codex CLI" models on the same benchmark: the harness hides the model
Specific Labs dropped Real-SWE, an enterprise-code SWE benchmark, and the leaderboard is a great study in why you should never read "Claude Code" or…
I Sell Memory APIs. I'm Also Building the Benchmark. Here's How I'm Trying Not to Rig It.
Hey everyone. This time I'll go through what got me started on this benchmark, and the core of how it's actually built. It started from reading comp…
I Tested Q4_K_M vs MXFP4 on the Same Laptop — The Supposedly-Faster New Format Lost
I tested two local LLMs in two different quantization formats on the same laptop, on the same prompt, three trials each. The result is the opposite…
ช่องว่าง 0.3% แต่ราคาต่าง 2 เท่า, อ่านตาราง Terminal-Bench 4.0 ให้เป็น
ช่องว่าง 0.3% แต่ราคาต่าง 2 เท่า, อ่านตาราง Terminal-Bench 4.0 ให้เป็น โดย Nokka (นก-กา), นักเขียนอิสระสายเทคโนโลยี ผู้เขียนบทความอธิบายเทคโนโลยีให้…
เมื่อ Benchmark โกหกคุณ, SWE-Bench ProMax กับคะแนนจริงที่โมเดลเก่งสุดทำได้แค่ 41.2%
เมื่อ Benchmark โกหกคุณ, SWE-Bench ProMax กับคะแนนจริงที่โมเดลเก่งสุดทำได้แค่ 41.2% โดย Nokka (นก-กา), นักเขียนอิสระสายเทคโนโลยี ผู้เขียนบทความอธิบา…
I Tested 4 LLM APIs for Coding Work — All 20 Runs Passed, But Speed Varies 10x
I tested 3 free LLM API providers and one paid Claude plan on the same 5 real coding tasks. All 20 runs passed on first attempt. The story isn't whe…
Tried `sponsors/wonderwhy-er`: A Practical MCP Server Review
Tried sponsors/wonderwhy-er : A Practical MCP Server Review sponsors/wonderwhy-er is an MCP server that gives Claude a controlled interface to devel…
AxonASP vs. Native IIS ASP: Performance Benchmarks and Engine Architecture
Legacy VBScript and Classic ASP are traditionally bound to the Windows native execution engine ( cscript.exe or w3wp.exe ). AxonASP replaces this de…
We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model.
We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row. The LangChain…
A LongMemEval-S number you can reproduce
We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no w…
I Tested GLM-5.3-Flash and Qwen3.8-Flash on 24 Real Tasks
I test-ran both of this week's open-weight flash models against 24 small, real workloads from an actual product stack — structured extraction, SEO m…
We tested "tokenize before you compress" against 452 configurations, and it mostly held up
A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text…
What should an MCP tool return? I ran 72 trials instead of arguing
There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, fo…
Benchmarking AI Coding Agents on Real Pull Requests
No synthetic puzzles, no contaminated suites: 25 tasks harvested from PRs merged in 2026 across five languages, graded by each project's own held-ou…
AI Daily Digest — August 1, 2026: ARC-AGI-3 Harness Discovery, EU AI Gigafactories, Devin SWE-1.7
🤖💻 AI Daily Digest — August 1, 2026 OpenAI Shows How Two Harness Settings Tripled ARC-AGI-3 Scores OpenAI published a rare technical deep-dive on Ju…
39,000 Torrents: The Bug My Green Benchmark Never Caught
[Torrent911] Inception.2010.TRUEFRENCH.1080p.BluRay.x264-YIFY That's a real filename. The goal: extract "Inception" and "2010" from it, query the TM…
MCPMark v2: InsForge on Sonnet 4.6
Originally published on the InsForge blog , written by Tony Chang (CTO & Co-Founder). Reposted here with permission. In December we published th…
OpenRouter vs Vercel vs LLMGateway Performance
Every AI gateway adds a hop between your app and the model. The question that matters is what that hop costs at the moment your user is staring at a…
Can You Beat an LLM? Building Humans vs. Humanity's Last Exam
Frontier models are crushing benchmark after benchmark...so I built a quiz to ask this simple yet humbling question: can a human still beat them? Hu…
The Same RTX 5090, but the GPU Sat Idle — a CPU-Bound Go Solver and the Case for L2 Cache
This is the second in a short series that benchmarks a single RTX 5090 by re-running published Go solvers — programs that don't just play Go but pro…