Choose Your Own Data-Analytic Adventure

Okay, we'll calculate effect size. Thus we need:

  • Alpha Level (usually .05)
  • Statistical Power (or 1 - Beta Level)
  • Sample Size

Alpha level is, WHEN the null hypothesis is true, in other words, WHEN the population effect size is zero, the probability that we will reject the null . Since it's bad to reject the null WHEN it is true, we want a low alpha level. We'll go with the customarily low alpha level of .05.

Statistical power is, WHEN the specified (non-zero) effect size is true, the probability that we will reject the null hypothesis. Since, it's good to reject the null WHEN it is false, we want a lot of statistical power. We'll go with the customary minimally high statistical power of .80.

What is your sample size?


We were thinking about 35 in each group.
Okay. So, now have everything we need to calculate effect size, right?
  • Alpha Level = .05
  • Statistical Power = .80
  • Sample Size = 70

All we need to do now is fire up R and enter three lines of code:


# great site: http://www.statmethods.net/stats/power.html
# install the add-on package for calculating statistical power
install.packages('pwr')
# load the add-on package
library(pwr)
# calculate your effect size!
pwr.r.test(n =70 , r = , sig.level =.05 , power =.80)

A correlation of .33 is the population effect size that will permit you to reject the null hypothesis in 80% of your samples of size 70.

Jacob Cohen gives guidelines for effect sizes:

  • "small" r = .1
  • "medium" r = .3
  • "large" r = .5

These guidelines are truly garbage, but they sound great in research proposals. Never use these guidelines in your deep data analytic work, but feel free to reference them in your write-up. As data analysts, we should sometimes throw our audience a bone... as long as we ourselves don't mistake the bone for meat! So, you can write, "Our sample size give us a reasonable chance (power = .80) of detecting a medium effect size (r = .33), which is adequate in light of the researchers' literature review and professional experience (Cohen, 1988)."

You decide to:
Let's call it "good" and submit this data analytic strategy.
or
Let's go at it from another angle and calculate the statistical power of your study.
or
Let's go at it from another angle and calculate the ideal sample size of your study.