Your eval's confidence interval assumes independent examples. Yours are clustered.
Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: e…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
aws
backend
beginners
career
claude
cybersecurity
database
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
An LLM judge is a biased instrument, not a measurement
Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Varian…
Gaussian Process Classification
Adapted from an appendix of my MS thesis. Classification We have considered regression problems where the targets are real valued. Classification prob…
When SuSiE Says '95% Confident', Is It?
Benchmarking the Honesty of Fine-Mapping Credible Sets Fine-mapping has a promise built into its output, and almost nobody checks whether the promise …
A model with R-squared near 0 can still give valid 90% prediction intervals - here's why (and the catch)
I recently calibrated a recovery-rate model that had only two weak features. Its point accuracy was almost nothing — R² basically zero. I expected its…
Conformal prediction silently breaks under drift - and how to make it hold
Conformal prediction is the easiest way to put a calibrated uncertainty band around any model: wrap a point predictor, and you get intervals with a fi…
Stop Shipping ML Models With Bare Floats: A Deep Dive Into Statistically Rigorous Model Evaluation
Stop Shipping ML Models With Bare Floats Every week, somewhere, a team makes a deployment decision that looks like this: Model A: AUROC = 0.847 Model …
Power analysis for LLM evals: how big does your eval set need to be to catch a 5% regression?
TL;DR: Most eval sets are sized by "what we had lying around", not by what they can actually detect. If your eval set is 50 traces and you are trying …
Graduate Statistics Problem Sets
I put my coursework from SIUe's Master's in Mathematics program up on the problem sets section of this site. Five courses from 2021-2022 that formed t…
Model Selection for Weibull Series Systems: When Simpler Models Suffice
When can you safely use a simpler model for a series system? I ran extensive simulation studies with likelihood ratio tests to get a quantitative answ…
I Built CausalLens — A Free, Open-Source Causal Impact Calculator for Time Series (5 Methods, Zero Setup)
I want to show you a tool I just open-sourced. It's called CausalLens, and it answers one specific question that most analytics stacks get completely …
Cointegration and Pairs Trading: When Time Series Move Together
Pairs trading rests on a simple idea: find two assets that move together, wait for them to diverge, and bet on convergence. The hard part is defining …
Understanding Correlation in PHP: Pearson vs Spearman vs Kendall Tau
Correlation helps you understand whether two variables move together and how strongly they are related. In this article, you'll learn how Pearson, Spe…
Lending's Old Faithful: How a 1958 Breakthrough Still Holds Off the AI Rush
Series: Building an Explainable AI Underwriter This is Part 2 . ↜ Part 1: Why Explainable AI Matters in Underwriting ↝ Part 3: Human Intervention Thre…