Lecture 2

Context

Suppose that, week before the 2020 presidential election, you did some polling.
- You use a list of the people registered to vote in Georgia.
- And you make phone calls.
- Each call, you select a voter uniformly at random from the list, e.g. by rolling a 7.23M -sided die.
- And then you ask the potential voter whether they plan to vote.
Suppose also that all these registered voters will
- Pick up the phone will called
- Respond honestly to your questions
- Not change their minds about voting
That is, suppose they tell us whether they do ultimately vote on election day.

Polling Results

You put your polling results in a table.
It has one column for each call.
In that columns, you record …
- the call number —a number from .
- the response of the person you called— for ‘yes and for no’.

You summarize the results with an extra column: the mean of the responses.
- Remember that the mean of a binary variable is a frequency. Or a proportion.¹
- You found that 68% of the people polled said they would vote.

Sample Versus Population

We want to estimate the proportion of all registered voters — our population — who will vote.
To do this, we use the proportion of polled voters — our sample — who said they would.
When the election occurs, we get to see who turns out to vote.
- 5.05M people, or roughly 70% of registered voters, actually vote.

Before the Election

Outcomes for our sample of polled voters.
To enumerate our sample, we …

give each call a number .
write for the turnout of the th person we called.

After the Election

Outcomes for the population of registered voters.
To enumerate our population, we …

give each registered voter a number .
write for the turnout of the person with ID .

Success!

Our sample proportion is close to the population proportion

Before the Election

After the Election

That’s pretty accurate. Our reputation as a turnout pollster is intact for now.
But unless we’re looking to retire, one success isn’t enough. We’re going to poll again.
If our methods aren’t reliable, we’ve got work to do fixing them.
Even if they are, if we overstate our accuracy, we’re going to have to answer for it.
- e.g. we could say we’ll be off by at most 2%, since that’s what happened this time.
- But if it’s 4% next time, that’s not going to look good.

Was it luck that we got as close as we did?
Could we have predicted how close we’d get before the election happened?

Review: Connecting Sample and Population

For each call , we randomly select a voter with an id we’ll call .
And we record as that call’s outcome the turnout of that voter: .
On each call, each registered voter has a chance of being called.
- This is called sampling with replacement because we could call the same person twice.
- In our poll, this is unlikely because we’re making a small number of calls relative to population size.

Our Poll

Our First Friend’s Poll

Our Second Friend’s Poll

The Sampling Distribution of our Estimate

The Sampling Distribution in R

We run 10,000 simulated polls and store who we call (J) and what they say (Y).

Js = array(dim=c(10000, n))
Ys = array(dim=c(10000, n))
for(rr in 1:10000) {
  Js[rr,] = sample(m, n, replace=TRUE)
  Ys[rr,] = y[Js[rr,]]
}

We calculate the sample proportion for each poll.

meanY.samples = rowMeans(Ys)

And we histogram the result.

ggplot() + geom_bar(aes(x=meanY.samples, y=after_stat(prop)), alpha=.2)

A Sketch
The Real Thing

Before we look at the real thing, let’s sketch the histogram we expect to see.

Checklist

Is the histogram you drew in the location you think it should be?
What about the shape? Is its spread what you think it should be?

Annotating the Sampling Distribution Histogram

There are a few features we may want to highlight.
- The mean of the sampling distribution is the solid blue line.
- The middle 2/3 of the sampling distribution lies between the dashed blue lines.
- The middle 95% of the sampling distribution lies between the dotted blue lines.
Our estimation target — the turnout frequency in the population — is drawn as a wide green line.
- It’s in exactly the same place as the solid blue line.²
- What does this tell us about our estimator?

Revisiting our Questions

Observation. Our estimate — the black dot — is close to our estimation target. It’s within 2%.

Q. Did we get lucky?

Not really.

In 68% of polls, the estimator is within 2% of the target.
In 95% of polls, the estimator is within 4% of the target.

Q. Could we have predicted how close we’d get before the election happened?

Yes, in sense.

We will use an interval estimate—a range of values the estimation target is likely to be in.
The width of this interval speaks to the ‘how close’ question.
The coverage probability — the probability our estimate is actually that close — qualifies this answer.

Interval Estimation

Our point estimate of the turnout frequency in our population is the turnout frequency in our sample: .
So let’s try an interval of width 0.02 centered on it: .
- This is just a width we chose arbitarily.
- Maybe it’s wishful thinking. Being off by at most 1% sounds good.
What we want is for our interval to cover our estimation target.
- i.e. for the population frequency to be in our interval.
This one doesn’t. Is that just bad luck? Or is it typical of intervals?
Let’s see what happens when our friends try intervals like this.

Interval Estimation: Our Friends’ Polls

Here are the interval estimates our first two friends.
That is, the interval estimates based on the pink and teal rows in our sampling distribution table.
One of these intervals covers the estimation target. The teal one.
So between ours and our two friends, we’re covering 1/3 of the time. Not great.
But that’s just three polls. To get a better sense of how often this happens, let’s do it for a hundred.

Interval Estimation: 100 Polls

cover the estimation target.
This gives us a sense of the probability that our interval covers the target.
If we want to be more precise, we could do the same for millions of different polls.
Let’s not. Instead, let’s find a more direct way to calculate the coverage probability.

Coverage Probability and Sampling Distributions

Activity. Explain how to calculate this coverage probability using the sampling distribution of your point estimate .

Suppose you want to use a diagram like this to count how many of our 100 intervals cover the estimation target.
But you don’t want to look at the horizontal segments for each poll. They’re small and hard too see.
You just want to use the dots representing each poll’s point estimate .
Sketch something on top of our diagram to help you count.

Calculating Coverage

Let’s shade in an interval of width .02 centered on the estimation target.
- This gives it ‘arms’ the same length as our interval estimates have.
- And its arms touch a point estimate if and only if the point estimates’ arms touch it.
That means we can count the intervals that cover by counting the point estimates between the dotted lines.
What, in terms of the sampling distribution of the point estimate, is the coverage probability?

The Coverage Probability

It’s the probability that a random draw from the sampling distribution lies in the green shaded area.

And it’s about 43%. We can get that by counting dots.

mean(mean(y) - .01 <= meanY.samples & meanY.samples <= mean(y) + .01)

[1] 0.4278

Or by finding the area of the histogram that’s shaded green.

Ad-hoc Interval Estimates

What we just did was choose a width and calculate a coverage probability.

The coverage probability we found — 43% — probably wasn’t what we wanted.
Think of how you’d advertise your polling services.
‘I’m right about half the time. Actually, a bit less than that’.
95% sounds a lot better. For that, we’ll have to use a wider interval.

Calibrated Interval Estimates

Instead of choosing a width and calculating the coverage probability, let’s go backward.
- We’ll choose a coverage probability — 95% is conventional.
- And we’ll calculate the width we need to get it.
An interval estimate calibrated like this—to have a given coverage—is called a confidence interval.
- Let’s think about how to do that. Again, we’ll use the sampling distribution of our point estimate.
- Let’s take a look at our annotated histogram of point estimates again.

Using the Sampling Distribution for Calibration

Question.

Suppose you and your friends want to draw 95% confidence intervals around your point estimates.
How wide do you have to make them to actually get 95% coverage?

Review—Our Annotations

The mean of the sampling distribution is the solid blue line—and is the same as the estimation target.
The middle 2/3 of the sampling distribution lies between the dashed blue lines.
The middle 95% of the sampling distribution lies between the dotted blue lines.

Using the Sampling Distribution for Calibration

Answer.

The width of a 95% interval should be the width between the dotted blue lines.
That’s the width of the ‘arms’ containing 95% of the estimates drawn from the sampling distribution.
You can check that these intervals have the coverage we want.

coverage = mean(mean(y)- dotted.width/2 <= meanY.samples & meanY.samples <= mean(y) + dotted.width/2)
coverage

[1] 0.9516

A Problem

We can’t calibrate intervals like this in real life.
- When we run our a poll, we get a single point estimate based on our sample.
- We don’t know the sampling distribution of this point estimate until election day.
But what we actually do is almost the same.
- We do the same thing.
- But we use an estimate of the sampling distribution in place of the thing itself.
That’s what we’ll talk about next class.