I audited 249 of my own AI coding sessions. The problem wasn't lying.
I never scrolled up I use a coding agent every day. I give it a task, go do something else, and come back to a line at the bottom of the screen like t…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
claude
cybersecurity
database
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
Testing a Two-Chart Compatibility Engine Without Inventing Missing Data
A compatibility engine can be deterministic and still be dishonest. The obvious failure is a wrong calculation. The subtler failure is unjustified cer…
Debugging a black box: 36 renders against Claude, and the part where my own data was wrong
If you've built an MCP App — the HTML widget an MCP server hands a host to render inline — you may have hit this: the tool call succeeds, structuredCo…
A Safe Outcome Can Hide a Failed Security Control
I kept coming back to a small problem in security testing: If the business outcome is safe, do we know that the control under test actually worked? Of…
Benchmark an AI Coding Task Across Mobile Backgrounding, Network Switches, and Battery Pressure
Most AI coding assistants are demoed on a desk: laptop plugged in, stable Wi-Fi, full attention. But the moment I approve or monitor a long-running AI…
How to Test an AI Agent Before Giving It Access to Your Files
How to Test an AI Agent Before Giving It Access to Your Files AI agent demos usually show the happy path: a prompt goes in, a polished result comes ou…
Benchmark Mobile AI Reconnect Recovery Across Lifecycle Transitions With a Server You Can Kill
Last week I was testing an AI chat feature on my iPhone 14 (iOS 17.5, React Native 0.74, Hermes). The flow looked fine on Wi-Fi. Then I walked into an…
Python Selenium Architecture and the Importance of Python Virtual Environments
Modern software development demands applications that are reliable, efficient, and capable of delivering an excellent user experience. As organization…
Understanding Python Selenium Architecture and the Importance of Python Virtual Environments
Modern software development demands applications that are reliable, efficient, and capable of delivering an excellent user experience. As organization…
A league season is not a long tournament: four product bugs my test suite could never catch
I built a football prediction game for the 2026 World Cup. A month long, 104 matches, one winner at the end. It worked, people played, nothing caught …
Why I Validate Angular Compatibility Using the Published npm Package (Not the Source Code)
I just published a new article on a lesson I learned while maintaining my open-source Angular library. I used to validate compatibility against my wor…
Your link checker thinks deleted Telegram bots are alive
Delete a Telegram bot and https://t.me/your_deleted_bot keeps returning HTTP 200 with a page that looks completely normal. Every link checker I know o…
How to Test AI Agents Without Calling More LLMs
AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is…
Your Agent's Memory Is a Markdown File. Let's Audit It.
Quick check: does your agent stack have a memory.md in it somewhere? An AGENTS.md ? A notes file the agent appends to when something seems worth keepi…
Debugging LLM Calls: How to Trace What Your AI Agent Actually Did
So I was messing around with a little side project, a chatbot that answers questions about the World Cup. Fun, low stakes, nobody's paycheck depends o…
Coordinate-based UI tests break. So we read the accessibility tree instead — from inside the simulator.
Every recorded mobile test I have ever inherited died the same way: someone moved a button. The recording said "tap at (340, 712)". The redesign moved…
I gave the same fabricated answer to RAGAS and DeepEval. One scored it 0.0. The other scored it 1.0
Here's an output from a RAG system asserting a pricing claim it was never given, for a question its context couldn't answer. I ran it past the two mos…
Testing Non-Deterministic LLM Pipelines in CI: A Contract-Based Approach
Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters you…
OpenEval: Why LLM Evaluation Needs a Standard Format
Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, In…
Testing AI Coding Agents Beyond Code Generation: A Real-World Benchmark
1. Introduction Most public demonstrations of AI coding agents begin with an empty directory and end when a visible feature works. That is useful, but…
I Wrote Integration Tests for My MCP Failure Library. Here's the Pattern That Caught 3 Hidden Bugs.
I'd been shipping fixes to my MCP failure library for weeks. Every release felt solid. Then I've written one real integration test - and three "fixed"…
Stop letting your AI agents hallucinate test failures
I've seen enough CI pipelines die in an infinite loop of 'fix, retry, fail' to last a lifetime. You know the pattern. An agent-driven QA process runs …
Solution to Challenge 2 - Data Grid Validation and CRUD Operations
Challenge #2 is done — here's my solution to Challenge 2 - Data Grid Validation and CRUD Operations First, the gaps in the user story 1) The delete op…
How to Actually Unit-Test Google Apps Script (Vitest, Mocks & Clasp CI)
The bug that finally made me set up tests was a tax rounding error that quietly shipped wrong totals for two weeks. It lived in a plain function — no …
The Last Error Is Not Always the Root Cause: Diagnosing Android Test Failures
An Android UI test times out. Maestro reports that an element is missing. A screenshot shows a blank or half-rendered screen. It is tempting to fix th…
Unknown Time Is Not Noon: Modeling Missing Temporal Data Without Inventing Facts
Missing data is not the same thing as a convenient default. That sounds obvious, yet temporal software regularly converts an empty time field into mid…
How do you measure something that gives a different answer every time?
I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC…
Selenium with Python: The Ultimate Guide to Web Automation Testing
Introduction : In today's fast-paced software development world, delivering high-quality applications quickly is a top priority. Manual testing is eff…
Your OpenAPI Spec Is Decoration Until CI Enforces It
My rule: if an OpenAPI spec is not wired into CI as an executable contract, it is decoration. A spec that no test asserts against will drift within tw…
My eval said a perfect MCP server was broken. It was the eval that was lying.
Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade , the first real run produced a result that looked like a scoop: conte…