Choose Your Own Data-Analytic Adventure

Okay, we'll calculate sample size. Thus we need:

  • Alpha Level (usually .05)
  • Statistical Power (or 1 - Beta Level)
  • Effect Size (i.e., the w statistic)

Alpha level is, WHEN the null hypothesis is true, in other words, WHEN the population effect size is zero, the probability that we will reject the null . Since it's bad to reject the null WHEN it is true, we want a low alpha level. We'll go with the customarily low alpha level of .05.

Statistical power is, WHEN the specified (non-zero) effect size is true, the probability that we will reject the null hypothesis. Since, it's good to reject the null WHEN it is false, we want a lot of statistical power. We'll go with the customary minimally high statistical power of .80.

What do you mean "specified (non-zero) effect size"?

That is the tricky question that we must tackle now. To calculate sample size, we've got to specify the effect size in the population.

But, if we knew the effect size in the population, why would we even conduct this research? That's exactly what we are trying to find out. Isn't it?

There lies the rub. We must guess, and then we use that guess (in conjunction with our alpha level and statistical power) to calculate sample size.

Guess?

Yes. Our guess should be fairly conservative, because if we guess too big of a population effect, then we'll collect too small of sample size.

How do I guess the effect size? Am I supposed to guess the chi-square statistic? Or, am I supposed to guess the proportions in each of the four groups: Anxious control? Calm control? Anxious intervention? Calm intervention?

Actually, that's very close to what we'll do. We'll do a combination of your two ideas. We'll calculate the w statistic:

w

Before you panic, let's break out some paper and pencil, and let's break down the problem into manageable chunks. The only tricky part is getting the population proportions for each of the four groups.

scratch1

Let's focus on the control group for starters. What percentage of control patients do you think will be clinically anxious as per your MYPAS cutoff score of 30?

Based on my experience... about 60%.

Great. That means 40% calm and 60% anxious for control patients. Right?

scratch2

Now, let's think about the intervention group. What percentage of intervention patients do you think will be clinically anxious as per your MYPAS cutoff score of 30?

Based on my experience... about 20%.

Great. That means 80% calm and 20% anxious for intervention patients. Right?

scratch3


Honestly, that's the hard part, but your deep substantive knowledge made it pretty easy. Now, we just need to work through the math. I'll fly through the math for now, and we can talk about it later if you want.

scratch4

You have a w of .41. Jacob Cohen gives guidelines for effect sizes:

  • "small" w = .1
  • "medium" w = .3
  • "large" w = .5
These guidelines are truly garbage, but they sound great in research proposals. Never use these guidelines in your deep data analytic work, but feel free to reference them in your write-up. As data analysts, we should sometimes throw our audience a bone... as long as we ourselves don't mistake the bone for meat! So, you can write, "We determined sample size based on a medium to large effect size (w = .41), which made sense in light of the researchers' professional experience (Cohen, 1988)."

Okay. So, now have everything we need to calculate sample size, right?

  • Alpha Level = .05
  • Statistical Power = .80
  • Effect Size = .41


All we need to do now is fire up R and enter three lines of code:
# great site: http://www.statmethods.net/stats/power.html
# install the add-on package for calculating statistical power
install.packages('pwr')
# load the add-on package
library(pwr)
# calculate your sample size!
pwr.chisq.test(w =.41, N = , df =1, sig.level =.05, power =.80)

Your sample size (N) is 46.7.

I suggest that you collect more 46.7 observations. Definitely don't try to collect exactly 46.7 observations, because you'll never find that .7 child! Collecting more observations does a couple of things. First, it protects you in case you were over-optimistic about your population effect size (w = .41). Second, if your population effect size was dead-on accurate, you will increase your statistical power by collecting more observations, and that's never a bad thing.

You decide to:
Let's call it "good" and submit this data analytic strategy.
or
Let's go at it from another angle and calculate the statistical power of your study.