Before the Election

• You want to estimate the proportion of all registered voters — the population — who will vote.
• To do this, you use the proportion of polled voters — your sample — who said they would.
  • You’re probably not going to match the population proportion exactly …
  • … so you report an interval estimate—a range of values we claim the population proportion is in.
  • You’re know you’re not going to be right 100% of the time you make with claims like this …
  • … so you state a nominal coverage probability—how often you’re right about claims like this.¹
• Without thinking too hard about it, you …
  • … report, as your interval, your sample proportion (72%) ± 1%. Because it sounds good.
  • … and say your coverage probability is 95%. Because that’s what everybody else says.

After the Election

• When the election occurs, we get to see who turns out to vote.
  • 5.05M people, or roughly 70% of registered voters, actually vote.
  • You — and future employers — can see how well you did. And how well everybody else did.
• Your interval missed the target. It doesn’t contain the turnout proportion.
  • Your point estimate is only off by 2%. But you overstated your precision.
  • Now you’re kicking yourself. You’d briefly considered saying ± 3% or ± 4%.
  • That would’ve done it. But that didn’t sound as good, so you went for ± 1%.
• You hope you’re not the only one who missed. So you check out the competition.

The Competition

• It turns out that everyone got their point estimate exactly like you did.
  • Each of them rolled their 7.23M-sided die 625 times.
  • And after making their 625 calls, they reported their sample proportion.
• You’re not the only one who missed, but you’re part of a pretty small club.
  • 5% of the other polls got it wrong.
• But you are the only one who claimed they’d get within 1%.
  • Everyone else said claimed they’d get within roughly 4%.
  • Most were right. About 95%. And a lot of them had worse point estimates than you.
• What you have is a calibration problem.

Question

• I’ve drawn the sampling distribution of an estimator, 100 draws from it as ●s, and 3 interval estimates.
• One of these interval estimates is calibrated to have exactly 95% coverage. Which is it?
  • A. The one around the ● on top.
  • B. The one around the ● in the middle.
  • C. The one around the ● on the bottom.
• Hint. You can reach your twin with your arms if your twin can reach you with theirs.
• Hint. Maybe you look like this ● and your twin like this |.

Calibration in Reality

• But you didn’t know any of this. You just knew your own point estimate.
• Without all that post-election information, you didn’t know how to calibrate an interval estimate.
• But everyone else did. Their widths were almost the same as the width you’d choose now.

Payoff

• To estimate this sampling distribution, you plug your point estimate \(\hat\theta\) into the Binomial formula. \[ \hat P\left(\sum_{i=1}^n Y_i = s\right) = \binom{n}{s} \hat\theta^{s} (1-\hat\theta)^{n-s} \quad \text{ estimates } \quad P\left(\sum_{i=1}^n Y_i = s\right) = \binom{n}{s} \theta^{s} (1-\theta)^{n-s} \]

• To calibrate your interval estimate, you …

  • use rbinom to draw 10,000 samples from this estimate of the sampling distribution.
  • Even remembering to divide by \(n\).
  • And use the function width from the Week 1 Homework to find an interval that covers 95% of them.

theta.hat = mean(Y)
samples = rbinom(10000, n, theta.hat) / n
interval.width = width(samples, alpha=.05)

• You nail it. Your interval covers the estimation target \(\theta\) just like 95% of your competitors’ do.

Looking Back on Your Success

• Remember your sucessfully calibrated interval estimate from a moment ago?
  • You did that by plugging your point estimate \(\hat\theta\) into the Binomial formula.
  • And you got a nicely calibrated interval estimate. That was great.
• But let’s take a closer look. Let’s compare your estimate of the sampling distribution to the actual thing.
  • We can do that because we have a bunch of draws from the real thing—all the others polls. ●s.
  • And if we want more draws, since it’s after the election, we can simulate as many polls as we want.
• It doesn’t look great. It’s off center.
  • Its mean is a bit higher than the population proportion \(\theta\).
  • It’s actually our sample proportion \(\hat\theta\). That makes sense.
• The mean of the Binomial distribution with success rate \(\theta\) is \(\theta\) and we’re using \(\hat\theta\) in its place.
• It turns out that this doesn’t matter much. It worked just fine for calibration. Why?

Calibration Comparison

• It doesn’t matter because we’re not putting arms on draws from the estimated sampling distribution.
• We’re putting arms on our point estimate³, which is a draw from the actual sampling distribution.
• To do this, we’re using the width—but not the center—of the estimated sampling distribution.
• And it works because that’s very close to the width we’d get from the actual sampling distribution.
• Above, I’ve drawn in shaded regions corresponding to two versions of the the population proportion’s arms.
  • The green region is the one we get from the actual sampling distribution. We can’t use this.
  • The red region is the one we get from the estimated sampling distribution. We do use this.
• And I’ve drawn in a version of our interval estimate calibrated each way. They’re almost the same.

Our Competitors’ Calibration

• Our competitors’ sampling distributions, just like ours, are centered on their sample frequencies \(\hat\theta\).
• But they’re all close to the width we get using the actual sampling distribution. You can see it above.
  • I’ve plotted the intervals they’d use — based on their sampling distribution estimates — in bold colors.
  • And the intervals they wish they could — based on the actual sampling distribution — more faintly.
• If you look very closely, you can see that …
  • the intervals around overestimates are slightly narrower than we’d want.
  • the intervals around underestimates are slightly wider than we’d want.
• But you have to look very closely. They’re all very close to the actual sampling distribution’s width.

All of Our Competitors

• We saw earlier that all of our competitors had almost-perfectly calibrated intervals.
  • They got their widths by plugging their sample frequencies \(\hat\theta\) into the Binomial formula.
  • And the result was almost exactly as if they’d plugged in the population frequency \(\theta\) instead.
• Let’s look again. This time, we’ll plot their estimated sampling distributions too.
  • But we’ll shift them all so they’re centered at zero.
  • That way we can compare their widths more easily. Because width is what matters.
• Each competitor gets their own color and we see their …
  • centered sampling distribution plotted as a line
  • centered interval estimate as a point with arms
• Compare to the actual sampling distribution (centered and shaded gray) and its middle 95% (dotted lines).

Why Does it Work?

• Why are we getting a good estimate of the sampling distribution?
• The Binomial distribution is continuous as a function of \(\theta\)—when \(\theta\) changes little, the distribution changes little.
• This means that, if we have a good estimate of \(\theta\), we have a good estimate of the sampling distribution.
• The relevant difference (after centering) is even smaller because the way the binomial changes is mostly location.
• Here I’m showing three estimates of the sampling distribution based on three point estimates \(\hat\theta\).
  • with centering (right) and without (left).
• In particular, point estimates at the center and two edges of the actual sampling distribution’s middle 95%.

The Estimate Works. Why Does it Work?

• You can think of this as a sort of ‘confidence interval’ for our estimate of the sampling distribution.
  • 95% of the time, you’ll get an estimate somewhere between the red and blue ones.
  • And, as a result, the width of your interval estimate will be somewhere between the red and blue widths.
• One way of looking at it is, when calibrate interval estimates this way, they’re almost perfectly calibrated.
  • Coverage may not actually be 95% but it’s very close.
  • You can see that the red interval does cover.
    • So will an interval around any point estimate between it and \(\theta\).
  • The blue interval doesn’t cover. It’s a fingernail too narrow.
    • But an interval around any estimate a fingernail smaller will cover.
    • It’ll be at least as wide and a fingernail closer.
• Another way of looking at it is that 95% of the time, your interval misses by at most a fingernail.
• This, or bit worse, is usually what’s meant when someone says ‘95% interval.’ Don’t expect perfect calibration.