Choose Your Own Data-Analytic Adventure

Okay, we'll calculate statistical power. Thus we need:

  • Sample Size
  • Alpha Level (usually .05)
  • Effect Size (i.e., the w statistic)

Sample size is the size of the sample that you propose to collect. In your e-mail to me last week, you said that you planned to collect about 70 observations, so we'll go with that sample size (N=70).

Alpha level is, WHEN the null hypothesis is true, in other words, WHEN the population effect size is zero, the probability that we will reject the null . Since it's bad to reject the null WHEN it is true, we want a low alpha level. We'll go with the customarily low alpha level of .05.

Statistical power is, WHEN the specified (non-zero) effect size is true, the probability that we will reject the null hypothesis. Since, it's good to reject the null WHEN it is false, we want a lot of statistical power.


What do you mean "specified (non-zero) effect size"?

That is the tricky question that we must tackle now. To calculate sample size, we've got to specify the effect size in the population.

But, if we knew the effect size in the population, why would we even conduct this research? That's exactly what we are trying to find out. Isn't it?

There lies the rub. We must guess, and then we use that guess (in conjunction with our alpha level and statistical power) to calculate sample size.

Guess?

Yes. Our guess should be fairly conservative, because if we guess too big of a population effect, then we'll collect too small of sample size.

How do I guess the effect size? Am I supposed to guess the chi-square statistic? Or, am I supposed to guess the proportions in each of the four groups: Anxious control? Calm control? Anxious intervention? Calm intervention?

Actually, that's very close to what we'll do. We'll do a combination of your two ideas. We'll calculate the w statistic:

w

Before you panic, let's break out some paper and pencil, and let's break down the problem into manageable chunks. The only tricky part is getting the population proportions for each of the four groups.

scratch1

Let's focus on the control group for starters. What percentage of control patients do you think will be clinically anxious as per your MYPAS cutoff score of 30?

Based on my experience... about 60%.

Great. That means 40% calm and 60% anxious for control patients. Right?

scratch2

Now, let's think about the intervention group. What percentage of intervention patients do you think will be clinically anxious as per your MYPAS cutoff score of 30?

Based on my experience... about 20%.

Great. That means 80% calm and 20% anxious for intervention patients. Right?

scratch3


Honestly, that's the hard part, but your deep substantive knowledge made it pretty easy. Now, we just need to work through the math. I'll fly through the math for now, and we can talk about it later if you want.

scratch4

You have a w of .41. Jacob Cohen gives guidelines for effect sizes:

  • "small" w = .1
  • "medium" w = .3
  • "large" w = .5
These guidelines are truly garbage, but they sound great in research proposals. Never use these guidelines in your deep data analytic work, but feel free to reference them in your write-up. As data analysts, we should sometimes throw our audience a bone... as long as we ourselves don't mistake the bone for meat! So, you can write, "We determined statistical power based on a medium to large effect size (w = .41), which made sense in light of the researchers' professional experience (Cohen, 1988)."

Okay. So, now have everything we need to calculate sample size, right?

  • Sample Size = 70
  • Alpha Level = .05
  • Effect Size = .41


All we need to do now is fire up R and enter three lines of code:

# great site: http://www.statmethods.net/stats/power.html
# install the add-on package for calculating statistical power
install.packages('pwr')
# load the add-on package
library(pwr)
# calculate your statistical power!
pwr.chisq.test(w =.41, N =70, df =1, sig.level =.05, power = )

Your statistical power is .93.


As a rule of thumb, we don't bother conducting a study unless the statistical power is greater than .80. By that standard, you are sitting pretty. Furthermore, your .93 (which is well above .80) gives you extra protection in case you were over optimistic about the population effect size. In general, the more power, the better.

You decide to:
Let's call it "good" and submit this data analytic strategy.
or
Let's go at it from another angle and calculate the ideal sample size of your study.