Bootstrapping Proportions

• When we’re estimating a proportion, the bootstrap sampling distribution is exactly the same as
  the parametric estimate we get by plugging the sample proportion into the Binomial distribution’s formula.
  • That’s good when we’ve sampled with replacement, so our sampling distribution is actually Binomial.
  • It’s less good when we’ve sampled without replacement, so our sampling distribution is Hypergeometric.
• Using this equivalence, we’ve talked a lot about how the bootstrap works in this special case.

Bootstrapping Differences in Proportions

• We have tried it out in the context of estimating a difference in proportions.
  • In particular, the difference in turnout between Black and non-Black voters in GA in the 2020 election.
• This difference does not have a sampling distribution with a simple parametric form, e.g. the binomial distribution.
• But we can calculate the bootstrap sampling distribution. That’s how we calibrated. It worked in this case.
  • The bootstrap samping distribution had almost the same width as the actual sampling distribtion.
  • And you can check, computationally, that bootstrap-calibrated intervals have almost exactly 95% coverage.
• When we sample with replacement, it usually works.
  • In fact, it’ll work for every estimator we talk about in this class.
  • We won’t have time to prove that in general—it does take some fairly sophisticated tools—but it is true.
  • But to give you a sense of why it usually works, I’ll take you through this case in next week’s homework.

An Important Limitation of the Bootstrap

• The bootstrap is a great method for understand what we have learned after we have data.
  • It’s easy to use and we usually get good calibration.
• But it’s also important to be able to reason about what we can learn before we have data.
  • For this, the bootstrap is not very helpful.
• Today, we’ll talk about that. To do it, we’ll introduce a new tool: normal approximation.
• And we’ll use it for an important ‘before data reasoning’ task: sample size calculation.
  • This is using what you know—or are willing to assume—to choose the size of your study.
  • In particular, to choose it so your confidence intervals are as narrow as you want them to be.

Normal Approximation

• A distribution’s normal approximation is a normal distribution with the same mean \(\theta\) and standard deviation \(\sigma\).
  • These show up everywhere because they’re easy to work with and the approximation tends to be good.
  • Approximation is good, in particular, for the distribution of a mean of independent random variables.
  • Or independent enough ones.
• Above, we see three Binomial distributions with their normal approximations:
  • the distributions of the mean of 10, 30, and 90 coin flips.
  • These approximations get increasingly accurate as the number of flips \(n\) increases.
  • That’s universal. It always happens with means. It’s called the Central Limit Theorem.

The Width of Normal Distributions

• One thing that’s convenient about the normal distribution is that it’s easy to reason about.
• In particular, it’s easy to reason about the width of its middle x%.
  • To include 68.3% of draws, you go out 1 standard deviation from its mean.
  • To include 95.4% of draws, you go out 2 standard deviations.
  • To include 99.7% of draws, you go out 3 standard deviations.
• To get almost exactly 95% of draws, we go out 1.96 standard deviations: \(\theta \pm 1.96\sigma\).
• Two is close enough in practice, but we tend to write 1.96 anyway.
  • It’s a signal about what we’re doing. You can get a 2 anywhere in a calculation.
  • When you see a 1.96, you know you’re talking about the middle 95% of the normal.

Calibration Using Normal Approximation

• Calibrating interval estimates using normal approximation is easy.
  • You estimate the standard deviation of your point estimator.
  • You go out ± 1.96 (estimated) standard deviations from your point estimate.

\[ \text{interval estimate} = \hat\theta \pm 1.96 \hat\sigma \]

• We’re choosing our interval’s width essentially the same way we always have.
  • We’re still using the middle 95% of an estimate of the sample distribution.
  • It just looks different because we have a convenient formula for that width.
• Above left, you can see two interval estimates superimposed on the sampling distribution of a sample proportion.
  • The red one is calibrated using the binomial as before.
  • The blue one is calibrated using normal approximation.
• Above right, you can see the binomial and normal sampling distribution estimates these are based on.

A Proportion when Sampling with Replacement

• When we sample from a binary population with replacement, our sample proportion’s distribution is Binomial.
  • And we can estimate that distribution by plugging our sample proportion into the Binomial formula.
  • But its normal approximation tends to be very good, so we can get away with estimating that instead.

\[ \begin{aligned} \text{normal approximation} & \ f_{\theta,\sigma}(x) \qfor \sigma^2 = \frac{\theta(1-\theta)}{n} \\ \text{corresponding estimate} & \ f_{\hat\theta, \hat \sigma}(x) \qfor \hat\sigma^2 = \frac{\hat \theta(1-\hat \theta)}{n}. \end{aligned} \]

• Pictured: three Binomial distributions with their normal approximations.
  • These are the distributions of the sample proportion when we draw a samples of size 10, 30, and 90 …
  • … with replacement from a binary population in which the proportion of ones is \(\theta=.5\).

A Proportion when Sampling without Replacement

• When we sample from a binary population without replacement, our sample proportion’s distribution is Hypergeometric.
  • And we can estimate that distribution by plugging our sample proportion into the Hypergeometric formula.
  • But again, its normal approximation tends to be very good, so we can get away with estimating that instead.
• When we do this, we can see where we go wrong when use use the Binomial—or equivalently the bootstrap.
  • When our sample size \(\color[RGB]{64,64,64}n\) is a meaningful fraction of our population size \(\color[RGB]{64,64,64}m\), the Binomial is too wide.
  • The Hypergeometric’s standard deviation differs from the Binomial’s by a factor of \(\color[RGB]{64,64,64}\sqrt{\frac{m-n}{m-1}}\).
  • When we’re sampling half our population, i.e. \(n=m/2\), that’s roughly \(\color[RGB]{64,64,64}\sqrt{1/2} \approx .7\).

\[ \begin{aligned} \text{normal approximation} & \ f_{\theta,\sigma}(x) \qfor \sigma^2 = \frac{\theta(1-\theta)}{n} \times \frac{m-n}{m-1} \\ \text{corresponding estimate} & \ f_{\hat\theta, \hat \sigma}(x) \qfor \hat\sigma^2 = \frac{\hat \theta(1-\hat \theta)}{n} \times \frac{m-n}{m-1}. \end{aligned} \]

• Pictured: three Hypergeometric distributions with their normal approximations.
  • These are the distribution of the sample proportion when we draw a samples of size 10, 30, and 90 …
  • … without replacement from a binary population twice the size in which the proportion of ones is \(\color[RGB]{64,64,64}\theta=.5\).

A Difference in Proportions

Using Normal Approximation

• What makes all this work is that we’re using good estimates of our estimator’s standard deviation.
  • If we have a formula for the estimator’s standard deviation, this is usuallly not so hard.
  • We estimate whatever population summaries show up in the formula and plug them in.
• But we do need to do a bit of work to get that formula. We’ll do that in a few minutes.
• This involves calculating expectations, so we’ll start there.

Properties of Expectations: Linearity

\[ \begin{aligned} E ( a Y + b Z ) &= E (aY) + E (bZ) \\ &= aE(Y) + bE(Z) \\ & \text{ for random variables $Y, Z$ and numbers $a,b$ } \end{aligned} \]

Properties of Expectations: Factorization of Products

\[ \mathop{\mathrm{E}}[YZ] = \mathop{\mathrm{E}}[Y]\mathop{\mathrm{E}}[Z] \qqtext{when $Y$ and $Z$ are independent} \]

The Standard Deviation of a Proportion

Sampling without Replacement

• When we sample without replacement, a lot of the same calculational steps still work.
• But the result we get is different, so we must’ve done something different. What’s different?

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\theta] = \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\frac1n \sum_{i=1}^n Y_i] = \mathop{\mathrm{E}}\qty[ \qty{ \frac1n \sum_{i=1}^n Y_i - \frac1n \mathop{\mathrm{E}}[\sum_{i=1}^n Y_i] }^2 ] = \frac{\theta(1-\theta)}{n} \times \frac{m-n}{m-1} \]

An Interval for Turnout in 2020

• Let’s compare our two approaches to calibration in our turnout poll.

\[ \begin{aligned} \textcolor{red}{\text{binomial interval}} &= 0.6800 \pm 0.0368 \\ \textcolor{blue}{\text{normal interval}} &= 0.6800 \pm 0.0366 \end{aligned} \]

• We have to go out to 4 digits, way beyond what’s statistically meaningful, to see any difference in these intervals.
  • Our interval estimate—either one—is telling you might be off by a few hundredths.
  • Who cares about another couple ten-thousandths at the edge of the interval?

Thinking Speculatively About Intervals

• Suppose we’re not satisfied with this level of precision, so we’re going to collect more data.
• Suppose we want our interval to be \(\pm .01\) instead of \(\pm 0.037\).
• How many people, in total, do we need to call? We can use the normal approximation to figure that out.

\[ \hat \theta \pm 1.96\sigma \qfor \sigma = \sqrt{\frac{\theta (1-\theta)}{n}} \qqtext{ is } \hat\theta \pm .01 \qqtext{if} 1.96\sqrt{\frac{\theta (1-\theta)}{n}} = .01 \]

• Now all we have to do is solve for \(n\). And, since we don’t know \(\theta\), use our best guess, \(\hat\theta=0.68\).

The Easy Version

• There’s a trick to this. Let’s compare the interval width we have to the one we want.

\[ \begin{aligned} \pm 0.037 &= \pm 1.96\sqrt{\frac{\hat\theta (1-\hat\theta)}{625}} \\ \pm 0.01 &= \pm 1.96\sqrt{\frac{\hat\theta (1-\hat\theta)}{n}} \end{aligned} \]

• All that changes in this formula is the same size. And the sample size ratio falls out of the interval width ratio.

\[ \frac{0.037}{.01} = \frac{1.96\sqrt{\frac{\hat\theta (1-\hat\theta)}{625}}}{1.96\sqrt{\frac{\hat\theta (1-\hat\theta)}{n}}} = \sqrt{\frac{n}{625}} \qqtext{ so } n = 625\left(\frac{0.037}{.01}\right)^2 \approx 8000 \]

• To get the new sample size, multiply …
  • the sample size we have
  • the square of the desired ratio of the interval widths.
• To double precision, quadruple the sample size. To triple it, increase it 9x.
• To get another digit, i.e. increase precision 10x, increase the sample size 100x.

Starting from Scratch

• What do we do if we don’t have any data yet?

  • Then we don’t have a ‘current interval width’ to compare to the ‘desired interval width’.
  • And we don’t have a sample proportion \(\hat\theta\) to plug in for the population proportion \(\theta\).

• But we can still use the formula we worked out earlier to get somewhere. \[ n = \frac{1.96^2 \ \theta (1-\theta)}{.01^2} \]

• We don’t have an estimate of \(\theta\), but we do know it’s between 0 and 1. And, consequently, so is \(\theta(1-\theta)\).

• So we know that if we just substitute \(1\) into our formula, we’ll get a number that’s bigger than we need.

\[ n < n' = \frac{1.96^2 \cdot 1}{.01^2} \approx 38400 \]

• That’s a bit excessive. In fact, we can substitute in \(1/4\) instead of \(1\). \[ n < n' = \frac{1.96^2 \cdot 1/4}{.01^2} \approx 9600 \]

• Much better. That’s pretty close to the number we got with preliminary data.

Why Can We Use 1/4?

• Here’s the claim. Why is it true? \[ n := \frac{1.96^2 \ \theta (1-\theta)}{.01^2} < n' := \frac{1.96^2 \ \times 1/4}{.01^2} \]

• Hint. Look at the graph of \(f(x) = x(1-x)\) above.

Why Continuous Distribution is a Blessing

• Think about what happens when you tried to compare binomial distributions for different sample sizes \(n\).
• If you’re lucky, and your sample sizess are all multiples of each other, then the probability shown in one wide bar
  gets split up into several narrow bars when sample size increases.
  • E.g., how the probability in the bar \(\color[RGB]{239,71,111}5/10\)] gets split into \(\color[RGB]{17,138,178}14/30\), \(\color[RGB]{17,138,178}15/30\), and \(\color[RGB]{17,138,178}16/30\) …
  • … and then into \(\color[RGB]{6,214,160}42/90 \ldots 48/90\) as sample size goes from \(\color[RGB]{239,71,111}10\) to \(\color[RGB]{17,138,178}30\) to \(\color[RGB]{6,214,160}90\).
  • That’s what’s going on in the plot on the left. What we see on the right is worse.
• If your sample sizes aren’t multiples, we don’t just split up one wide bar’s probability into several narrow ones.
  • For the narrow bars we see straddling two wide bars, we have to ‘merge’ probability from two wide bars.
  • It’s a mess. And using bars hides the worst of it.
  • Most of the points that have mass for one \(n\) will have none for the others.
  • It’s not possible to have a sample mean of \(5/10\) with a sample size of \(25\). It’s the wrong denominator.
    • Probability mass at \(\hat\theta = 1/2\) can go from maximal for \(n=10\) to zero for \(n=25\) …
    • … even if the population mean is the same: always \(\theta=1/2\). That happens in the right plot.
• Using an approximation with zero mass at any particular point, like the normal distribution, lets us avoid all this.
  • We do have to integrate whenever we want to calculate a probability.
  • But we can easily compare the probabilities of the same interval at different sample sizes.
  • In a sense, all the ‘splitting’ and ‘merging’ is built into the approximation.