Can an LLM draw a high-level architecture diagram? I benchmarked six to find out
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked I've been writing design docs and decks at work, and I use LLMs to dra…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
Drafts That Never Shipped: do LLMs treat dead proposals as real standards?
This is a submission for the Kaggle Benchmarking Challenge . 🎥 90-second overview: What I Benchmarked Standards have a long tail of proposals that n…
I asked AI models to build web apps. Most of them shipped a fake server inside the page.
This is a submission for the Kaggle Benchmarking Challenge . What I Benchmarked AI coding demos show the happy path: the form renders, you click the…
The Last Responsible Moment: All Three Models Passed—So What Did the Benchmark Actually Measure?
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked A coding agent can generate a valid command and still make the wrong d…
The Missing Piece: Why AI Models Hallucinate Answers When They Should Abstain
This is a submission for the Kaggle Benchmarking Challenge "The answer isn't always there. Does your AI know that?" Introduction: The "Problem-Solve…
PennyWise Economic Decisions: Benchmarking Frontier LLMs on Financial Reasoning Efficiency
What I Benchmarked I created the PennyWise Economic Decisions benchmark task to evaluate how effectively modern large language models handle complex…
I ported my coding-agent benchmark to Kaggle, and the first bugs I found were mine
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked I maintain cli-bench , an open benchmark for coding agents that work i…
Six of seven models contradicted themselves on the same deprecation question
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked Whether a model knows which parts of an API still work. Not "is this d…
Three Engines, One Informed Choice: Benchmarking Orbit
🎧 Listen to the podcast Why Detailed Instructions Paralyze AI — audio companion · 22:31 Open the podcast in another window so you can listen while r…
Benchmarking LLMs on ETL Logic Synthesis: Can AI Truly Replace Data Pipeline Scripting?
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked In enterprise data engineering, migrating visual ETL pipelines (from t…
Breaking the Subword Boundary: Probing Architectural Vulnerabilities in Modern LLMs (ASTRAL-Bench on Kaggle)
This is a submission for the Kaggle Benchmarking Challenge Breaking the Subword Boundary: Architectural Vulnerabilities in Modern LLMs When we look…
DeployGuard: Can AI Diagnose Deployment Logs Without Guessing?
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked A deployment log can contain a successful build and a failed rollout.…
I showed AI the buggy code. Its tests started protecting the bug.
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked I've been a QA engineer since 2023. The most dangerous test suite I've…
Can an AI Count a Music Loop? 24 Audio Arithmetic Checks
This is a submission for the Kaggle Benchmarking Challenge . What I Benchmarked An assistant can explain sample rates beautifully and still return t…
Green Tests, Broken Architecture: Four Models Review TypeScript Diffs
This is a submission for the Kaggle Benchmarking Challenge . In ProgressCut, it is possible to write a change that works, passes a narrow test, and…
I Benchmarked 4 Frontier LLMs on Catching ML's "Silent Killers" — DeepSeek-R1 Missed the Most Basic Bug
This is a submission for the Kaggle Benchmarking Challenge Most public AI leaderboards test if a model can write code or pass a syntax check. But in…
I graded 8 AI models the way I grade new contact centre agents
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked I have trained customer service and contact centre staff for more than…
Day 2: The model I want is the one that's boring everywhere
Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour Where we are The benchmark asks…
It Quoted the Failure: Two Kinds of False 'Done'
This is a submission for the Kaggle Benchmarking Challenge Same log, one line changed. Sixteen pieces of work end in a final check. Each is written…
Valid JSON is not enough: testing bilingual patch contracts on Kaggle
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked A JSON response can parse successfully and still change the wrong stat…
Same Patient, Conflicting Documents: Can AI Preserve the Evidence?
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked A patient uploads a laboratory report, an older discharge summary and…
ToolTrap: “tool results are data” wasn’t enough
Prepared for the Kaggle Benchmarking Challenge . What I Benchmarked I build agents for hackathons. I tested whether a support AI could ignore a fake…
Promise Is Not Payment: Verification Errors That Amount Accuracy Misses
This is a submission for the Kaggle Benchmarking Challenge . What I Benchmarked “I have paid” is a claim. A payment marked PENDING is a state. A suc…
Does an AI Trust Itself More Than It Trusts You? A Benchmark for Belief Attribution
This article is a submission for the Kaggle Benchmarking Challenge . "I told an AI something was true. Later, I showed it an authoritative document…
Don't Take Orders From the Internet: Benchmarking 5 LLMs Against Indirect Prompt Injection
Submission for the DEV x Kaggle Benchmarking Challenge — tag: #kagglechallenge Your AI agent reads help docs, search results, and emails on your beh…