Statistical Power & Sample Size

Overview

Power analysis plans the probability of rejecting a specified null under an assumed alternative. It answers how many independent experimental units are needed to detect a scientifically important effect, or what effects a feasible sample could detect. Choose the inferential goal first: precision, equivalence, noninferiority, or sequential monitoring need their own calculations; the bundled superiority-test helpers do not cover them.

Four quantities are locked together for any given test: sample size (n), effect size, significance level (α), and power (1 − β). For a fixed design, analysis, and nuisance parameters, fixing three permits solving for the fourth when a solution exists. Every calculation in this skill is some rearrangement of that relationship.

This skill covers the two ways to do power analysis:

• Analytical power calculations (exact under some standard-test assumptions; asymptotic approximations for others) — see references/closed_form_recipes.md.
• Simulation / Monte Carlo (requires a credible data-generating process and a calibrated planned analysis) — see references/simulation_based_power.md.

For choosing and converting effect sizes — usually the hardest part — see references/effect_sizes.md.

When to Use This Skill

• Determining required sample size before collecting data (a priori power analysis)
• Finding the minimum detectable effect (MDE) for a fixed, already-determined sample size
• Producing power curves (power vs. n, or power vs. effect size) for a grant or protocol
• Justifying a sample size for an IRB submission, grant, or pre-registration
• Powering designs with unequal group sizes or non-1:1 allocation
• Planning complex mixed models, GLMs, clustering, survival, mediation, and interactions through simulation when simpler approximations do not fit
• Accounting for multiple comparisons, attrition/dropout, or clustering in the sample-size estimate

Installation

The local examples were checked on Python 3.13 with statsmodels 0.15.0, SciPy 1.18.1, NumPy 2.5.3, pandas 2.3.3, and Matplotlib 3.11.2. Use an environment separate from the repository's development environment:

uv venv --python 3.13 .venv-power
uv pip install --python .venv-power/bin/python "statsmodels==0.15.0" "scipy==1.18.1" "numpy==2.5.3" "pandas==2.3.3" "matplotlib==3.11.2"
# Optional comparison / survival methods (also checked for current API use):
uv pip install --python .venv-power/bin/python "pingouin==0.7.0" "lifelines==0.30.3"

On Windows use .venv-power/Scripts/python.exe. Lifelines 0.30.3 requires pandas<3; the tested pin above accommodates it. Mixed models and GLMs are included in statsmodels. Record versions and seeds with the output; numerical smoke tests do not establish a study's effect assumptions or Type I error control.

The one decision that drives everything: the effect size

Power calculations are only as trustworthy as the effect size you feed them. Do not invent a number. Use, in rough order of preference:

1. A minimally important effect — the smallest effect that would actually change a decision or matter scientifically/clinically (the "smallest effect size of interest", SESOI). This is the most defensible basis: you power to detect what matters, not what you hope to see.
2. A pilot or prior-study estimate, with uncertainty and selection bias considered. Small pilots are imprecise; publication or significance-based selection can inflate effects. Use a justified uncertainty model or sensitivity range instead of an arbitrary shrinkage factor.
3. A convention (Cohen's small/medium/large) only as a last resort, and say so explicitly.

Whatever you pick, run a sensitivity analysis: report how required n changes across a plausible range of effect sizes, not a single point. A power analysis presented as one number hides its biggest source of uncertainty. See references/effect_sizes.md for benchmarks and conversions between d, f, r, η², odds ratios, and Cohen's h/w.

…