Inference & Testing

Read these first

1. 01p-Hacking and the Garden of Forking Paths

   Search hard enough and noise will hand you a beautiful result. This is how selection manufactures significance, why it happens even to honest analysts who ran only one test, and how to price the search you actually did.

2. 02Statistical Power and Sample Size

   Power is the resolution of your experiment, the chance you would actually notice a real effect if one were there. Work it out before you collect data, or you will spend months proving only that your instrument was too blunt to see anything.

3. 03The Likelihood Ratio Test

   A single rule for judging a simple explanation against a more elaborate one. Measure how much better the elaborate story fits the data you actually saw, then check whether that improvement is more than an extra free knob buys you for nothing.

4. 04The Neyman-Pearson Lemma

   The answer to a very practical question: given a fixed budget of false alarms, which test catches the most real signals? Rank the evidence by a likelihood ratio and act on the top of the list, nothing else can do better.

Then the rest

1. 05Analysis of Variance and F-Tests
2. 06Bonferroni and Holm Corrections
3. 07Effect Size and Cohen's d
4. 08Family-Wise Error Rate
5. 09Jarque-Bera and Normality Testing
6. 10Minimum Detectable Effect
7. 11One-Sample and Two-Sample t-Tests
8. 12Proportion Tests and Binomial Inference
9. 13The Chi-Square Goodness-of-Fit Test
10. 14The Duality of Tests and Confidence Intervals
11. 15The Wald Test
12. 16Always-Valid Inference and e-Values
13. 17Group Sequential Designs and Alpha Spending
14. 18The Score (Lagrange Multiplier) Test
15. 19Uniformly Most Powerful Tests