Sample-size and statistical power calculations for planning studies. Use whenever someone asks "how many subjects/samples/replicates do I need", wants an a priori power analysis, a minimum detectable effect (MDE), a power curve, or needs to justify a sample size for a grant, IRB protocol, or pre-reg

Statistical Power & Sample Size Overview Power analysis plans the probability of rejecting a specified null under an assumed alternative. It answers how many independent experimental units are needed to detect a scientifically important effect , or what effects a feasible sample could detect. Choose the inferential goal first: precision, equivalence, noninferiority, or sequential monitoring need their own calculations; the bundled superiority test helpers do not cover them. Four quantities are locked together for any given test: sample size (n) , effect size , significance level (α) , and power (1 − β) . For a fixed design, analysis, and nuisance parameters, fixing three permits solving for the fourth when a solution exists. Every calculation in this skill is some rearrangement of that relationship. This skill covers the two ways to do power analysis: Analytical power calculations (exact under some standard test assumptions; asymptotic approximations for others) — see references/closed form recipes.md . Simulation / Monte Carlo (requires a credible data generating process and a calibrated planned analysis) — see references/simulation based power.md . For choosing and converting effect sizes — usually the hardest part — see references/effect sizes.md . When to Use This Skill Determining required sample size before collecting data (a priori power analysis) Finding the minimum detectable effect (MDE) for a fixed, already determined sample size Producing power curves (power vs. n, or power vs. effect size) for a grant or protocol Justifying a sample size for an IRB submission, grant, or pre registration Powering designs with unequal group sizes or non 1:1 allocation Planning complex mixed models, GLMs, clustering, survival, mediation, and interactions through simulation when simpler approximations do not fit Accounting for multiple comparisons, attrition/dropout, or clustering in the sample size estimate Installation The local examples were checked on Python 3.13 with statsmodels 0.15.0, SciPy 1.18.1, NumPy 2.5.3, pandas 2.3.3, and Matplotlib 3.11.2. Use an environment separate from the repository's development environment: On Windows use .venv power/Scripts/python.exe . Lifelines 0.30.3 requires pandas<3 ; the tested pin above accommodates it. Mixed models and GLMs are included in statsmodels. Record versions and seeds with the output; numerical smoke tests do not establish a study's effect assumptions or Type I error control. The one decision that drives everything: the effect size Power calculations are only as trustworthy as the effect size you feed them. Do not invent a number. Use, in rough order of preference: 1. A minimally important effect — the smallest effect that would actually change a decision or matter scientifically/clinically (the "smallest effect size of interest", SESOI). This is the most defensible basis: you power to detect what matters, not what you hope to see. 2. A pilot or prior study estimate , with uncertainty and selection bias considered. Small pilots are imprecise; publication or significance based selection can inflate effects. Use a justified uncertainty model or sensitivity range instead of an arbitrary shrinkage factor. 3. A convention (Cohen's small/medium/large) only as a last resort, and say so explicitly. Whatever you pick, run a sensitivity analysis : report how required n changes across a plausible range of effect sizes, not a single point. A power analysis presented as one number hides its biggest source of uncertainty. See references/effect sizes.md for benchmarks and conversions between d, f, r, η², odds ratios, and Cohen's h/w. Avoid post hoc ("observed") power. Computing power from the effect size you just estimated is circular: for standard tests it largely restates the test statistic/p value and adds no independent evidence of adequacy. If a study is already done and you want to know what it could have detected, report a sensitivity analysis (MDE at the achieved n) or, better, the confidence interval around the observed effect. This is a common reviewer complaint — do not produce observed power even if asked without flagging the issue. Quick recipes (closed form) The bundled scripts/power.py wraps statsmodels and SciPy into one consistent interface so you don't have to remember which solver belongs to which test. Run from skills/statistical power/scripts/ or add that directory to sys.path . For two sample tests the return is n1 , with n2 = ceil(ratio n1) ; ratio=n2/n1 . Recheck power using the realized integer ratio. ANOVA rounds total n to a multiple of k groups ; paired n counts pairs. One sided alternatives use signed effects and "larger" / "smaller" . Proportion MDEs return signed Cohen's h and need a baseline to convert to feasible probabilities. Supported test= values: t ind (two independent means), t paired / t one (paired or one sample mean), anova (one way), two proportions , one proportion , correlation , chi2 (goodness of fit / contingency via effect size w ), linear regression (R² increment / f²). Full argument tables and the underlying statsmodels calls are in references/closed form recipes.md . When analytical assumptions do not fit: simulate Use an analytical method when its design and assumptions match the planned analysis. For logistic/Poisson regression, mixed effects / repeated measures models, cluster randomized trials, survival analysis, mediation, or multi way interactions , simulation is often useful when available approximations omit material design features. The logic is always the same three steps: 1. Simulate a dataset from your assumed truth (the effect you want to detect, plus realistic noise, baseline rates, cluster structure, etc.). 2. Analyze it with the exact test/model you plan to use on the real data. 3. Repeat many times (≥1,000; 5,000–10,000 for a stable estimate near 80%). Power is the fraction of replicates in which the test is significant. scripts/simulate power.py provides a reusable harness plus worked examples (two group difference, logistic regression, cluster randomized trial with an ICC, and a linear mixed model). The core is just: The callback must return a boolean rejection decision using the planned alpha internally. The harness's alpha argument does not threshold returned p values or pass alpha into the callback; returning a raw p value now raises TypeError instead of counting a nonzero float as rejection. The harness rejects nonpositive sample/replicate counts and invalid search bounds; an unmet target at the sample size cap raises an explicit error. The examples check convergence and finite p values. Expected failures raise SimulationFitError ; the harness counts them as non rejections and reports n failures / failure reasons , retaining every replicate in the denominator. It also reports warning counts; unexpected errors propagate. First check Type I error under the null. A noisy bisection search yields a candidate n: verify nearby sizes with more replicates and a fresh seed. Report the Monte Carlo confidence interval on the estimate (the harness returns it) to quantify simulation sampling error; it does not cover uncertainty in the assumed effect, model, or adaptively chosen n. See references/simulation based power.md for the full patterns, including how to search for the n that hits target power and how to model dropout and clustering. Adjustments people forget These routinely make the difference between an adequately powered study and an underpowered one. Apply them explicitly and state that you did. Multiple comparisons. If the analysis tests m hypotheses with a Bonferroni style correction, power each test at the corrected α (e.g. α/m), which raises n. Better: power on the family wise or FDR controlled procedure directly via simulation. Specify which primary, co primary, secondary, or interaction claims need adequate power and which multiplicity procedure applies; secondary endpoints need not all be confirmatory. Attrition / dropout / unusable samples. Power gives the n you need analyzed . Inflate the enrolled n: n enroll = ceil(n analyzed / (1 − dropout rate)) . A 20% dropout rate means enrolling 25% more than the formula returns. Clustering (design effect). For equal size parallel clusters with exchangeable correlation, DEFF = 1 + (m − 1)·ICC is a planning approximation. It is not a universal adjustment for repeated measures contrasts, unequal cluster sizes, or few clusters. Model those structures directly. Treating clustered data as independent is pseudoreplication and badly overstates power — for cluster randomized designs, simulate instead. One vs. two sided. Two sided is the default and almost always the right choice; a one sided test buys power only by refusing to detect an effect in the unexpected direction. Justify any one sided test. Unequal allocation. Equal groups are most efficient for the equal variance, equal cost two mean design used here; different costs or variances can change the optimum. If allocation is fixed by design (e.g. 2:1 treatment:control), pass ratio= so the calculation reflects it. Workflow 1. State the design and the planned analysis. Define the estimand, independent experimental unit, allocation, direction, nuisance assumptions, and exact planned analysis; choose a matching analytical method or simulation. 2. Choose the effect size on a defensible basis (SESOI shrunk pilot convention) and write down the justification. 3. Set α and target power. Conventional defaults are α = 0.05 (two sided) and power = 0.80; 0.90 is common for confirmatory/clinical work. State them. 4. Compute with scripts/power.py (closed form) or scripts/simulate power.py (simulation). 5. Sensitivity analysis. Recompute across a range of plausible effect sizes and produce a power curve. This is the deliverable, not a single number. 6. Apply adjustments for dropout, clustering, and multiplicity. 7. Report following the template below. Reporting template A defensible power statement contains every input, so a reader could reproduce it. Adapt: The numerical example above does not establish that d = 0.50 is clinically important. For simulation: also state the data generating assumptions (baseline rate, residual SD, ICC, cluster sizes), the number of simulations, and the Monte Carlo CI. Common pitfalls 1. Inventing the effect size or copying an inflated pilot estimate — the most common way power analyses go wrong. 2. Reporting a single n instead of a sensitivity range / power curve. 3. Post hoc / observed power — circular and uninformative; use sensitivity analysis or the effect size CI instead. 4. Ignoring clustering (pseudoreplication) — counting cells/measurements as if they were independent subjects. 5. Forgetting dropout — powering the analyzed n but enrolling the same number. 6. Confusing α with power , or one sided with two sided. 7. Powering only the primary endpoint while reporting secondary/interaction tests that need far larger n. 8. Using a t test formula for a model you won't actually fit (e.g. planning a logistic regression with a means based calculation) — match the power method to the planned analysis. Resources Scripts scripts/power.py — unified closed form interface ( sample size , power , mde , power curve ) over statsmodels/SciPy for all standard tests. scripts/simulate power.py — Monte Carlo power harness with simulate power() and find sample size() , plus worked examples (two group, logistic regression, cluster randomized, linear mixed model). References references/closed form recipes.md — per test argument conventions and tested statsmodels/Pingouin calls, including proportions, chi square, and regression. references/simulation based power.md — full simulation patte

statistical-power