Backtest a betting idea before you trust it
Originally published at quantplus.win . Sports betting backtesting software runs a betting rule over past games using only the prices and informatio…
Testing & QA
⚑ Report a ProblemLatest Testing & QA news from Tech News
All topics
agents
ai
api
architecture
automation
backend
beginners
career
claude
cybersecurity
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
node
opensource
performance
productivity
programming
python
security
showdev
softwareengineering
testing
tutorial
typescript
webdev
Сага о Дельта‑методе, часть 1: как оценивать ratio метрики?
Допустим, мы запустили АБ‑тест, рандомизировали клиентов, и теперь хотим оценить эффект фичи на средний чек. Средний чек — это пример rati…
Measuring How Cost Scales by Counting Instead of Timing
Code: Megapixel99/countfn Every empirical complexity tool I could find on either registry measures elapsed time. PyPI's big-O estimates the class fr…
A Curve Fitter That Refuses to Answer
Code: Megapixel99/undetermined Plenty of libraries fit a curve to measurements and hand you back a number. undetermined hands back a number with the…
Correct score, BTTS and Over/Under probabilities with Dixon-Coles: what I learned building it for MLS and Liga MX
Most football prediction pages give you three numbers — home, draw, away — and no way to check where they came from. I wanted the opposite: one mode…
Percentiles, the IQR and the 1.5 Outlier Rule: How to Flag a Bad Row
By Michael Nocito , data analyst · Published August 9, 2026 By the end of this page you can compute quartiles by hand, build the standard outlier fe…
What Is a P-Value? Worked by Shuffling Nine Real Orders 126 Ways
By Michael Nocito , data analyst · Published August 9, 2026 By the end of this page you can say what a p-value measures in one sentence, compute one…
Choice Leaks: I Tried to Generate 100 Random Digits and Failed in Four Measurable Ways
This morning a friend handed me a test for consciousness over Telegram. It was three sentences long and it had my name on it. If a thing has conscio…
Correlation Is Pairwise. Multicollinearity Isn't.
Almost every regression project I've seen in the last decade starts the same way. Load the data, df.corr() , plot the heatmap, scan for red squares,…
Running Multiple A/B Tests at Once: When Overlap Is Safe
Every experimentation program hits this question within its first few months: test A is running on the homepage, the checkout team wants to launch t…
Checking your A/B test every morning makes it lie to you
You launch the test on Monday. You check the dashboard each morning, because of course you do. On Thursday it reads p = 0.04, you call it, and you s…
Multi-Armed Bandit Testing: How It Works and When to Use
A multi-armed bandit is an algorithm that decides how to split traffic across several variations while a test is still running, shifting more visito…
How to Choose a Minimum Detectable Effect (MDE)
You have opened your sample-size calculator, entered your baseline conversion rate, kept the defaults for significance and power, and now the calcul…
You Don't Need a Math PhD for Data Science — You Need to Stop Skipping the Boring Step
Ask someone switching into data science what worries them and it's almost always the same thing: do I know enough math? So they spend weeks on gradi…
I read the metric libraries of five widely-used eval tools. The metric was never the hard part.
Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. T…
Your eval monitor fired on four days this week. At your sample size, that was the most likely count
A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dash…
Part 6 - STATISTICS
Bernoulli, Binomial Distribution & Central Limit Theorem Explained from First Principles When I first started learning probability, I came acros…
Your A/B test has three goals and they disagree. Now what?
Every A/B testing tutorial ends the same way: run the test, wait for significance, ship the winner. Then you run a real test and variant B converts…
Your eval's confidence interval assumes independent examples. Yours are clustered.
Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption:…
An LLM judge is a biased instrument, not a measurement
Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Vari…
Gaussian Process Classification
Adapted from an appendix of my MS thesis. Classification We have considered regression problems where the targets are real valued. Classification pr…
When SuSiE Says '95% Confident', Is It?
Benchmarking the Honesty of Fine-Mapping Credible Sets Fine-mapping has a promise built into its output, and almost nobody checks whether the promis…
A model with R-squared near 0 can still give valid 90% prediction intervals - here's why (and the catch)
I recently calibrated a recovery-rate model that had only two weak features. Its point accuracy was almost nothing — R² basically zero. I expected i…
Conformal prediction silently breaks under drift - and how to make it hold
Conformal prediction is the easiest way to put a calibrated uncertainty band around any model: wrap a point predictor, and you get intervals with a…
Stop Shipping ML Models With Bare Floats: A Deep Dive Into Statistically Rigorous Model Evaluation
Stop Shipping ML Models With Bare Floats Every week, somewhere, a team makes a deployment decision that looks like this: Model A: AUROC = 0.847 Mode…
Power analysis for LLM evals: how big does your eval set need to be to catch a 5% regression?
TL;DR: Most eval sets are sized by "what we had lying around", not by what they can actually detect. If your eval set is 50 traces and you are tryin…
Graduate Statistics Problem Sets
I put my coursework from SIUe's Master's in Mathematics program up on the problem sets section of this site. Five courses from 2021-2022 that formed…
Model Selection for Weibull Series Systems: When Simpler Models Suffice
When can you safely use a simpler model for a series system? I ran extensive simulation studies with likelihood ratio tests to get a quantitative an…
I Built CausalLens — A Free, Open-Source Causal Impact Calculator for Time Series (5 Methods, Zero Setup)
I want to show you a tool I just open-sourced. It's called CausalLens, and it answers one specific question that most analytics stacks get completel…
Cointegration and Pairs Trading: When Time Series Move Together
Pairs trading rests on a simple idea: find two assets that move together, wait for them to diverge, and bet on convergence. The hard part is definin…