Sampling

$$ \newcommand{X}{} \newcommand{Y}{}

$$

Surveys

What A Population Looks Like

\[ y_1 \ldots y_m \]

What We’re Estimating

• … is the proportion of these people who prefer B, i.e. who have \(y_j=1\).
• This could be pretty easy because we’ve just looked at all the ys.
• But we’re going to make it hard for ourselves by acting as if we haven’t.

\[ \text{estimand} = \frac{1}{m}\sum_{j=1}^m y_j \]

Estimating It

• Let’s assume we’ve been given a sample. Some list of preferences.
• This one. I’ve taken my plot of the population and colored in the dots I talked to.

\[ Y_1 \ldots Y_n \]

\[ \text{estimator} = \frac{1}{n}\sum_{i=1}^n Y_i \]

• The horizontal line shows the mean of my sample \(Y_1 \ldots Y_n\).
• That is, the fraction of times I head ‘I prefer B’ when I ask someone’s preference.

There are many ways to get a sample. How you do it matters.

Convenience Sampling

Sampling with Replacement

Sampling without Replacement

Which do you think works best?

• Which estimate do you think is closest to our estimand?
• Why?

Fake Data Simulation

• We can use fake data simulation to help us answer these questions.

  1. We make up a fake population.
  2. Run a fake (i.e. simulated) study on them exactly like we’re planning to do.
  3. Compare the resulting estimate to the thing we wanted to estimate.

• Because it’s fake, we get to poll the whole population so …

  1. We know what the right answer is. If we don’t, we don’t have a well-defined question.
  2. We can run the study over and over again to look at the distribution of estimates.

• If we’re not satisfied with that distribution, that’s evidence it’s a study we shouldn’t be running.

  • Once we’ve done this for a few study designs and seen what works — or at least what doesn’t,…
  • … we can choose one to use on our actual population. One that, if you’ve done this right, hasn’t been polled yet.

Which won? Was it luck?

Experiments

\[ \text{estimator} = \frac{\sum_{j:W_j=1} y_j}{\#\{j:W_j=1\}} - \frac{\sum_{j:W_j=0} y_j}{\#\{j:W_j=0\}} \]

1. Their coin. 1 for heads.
2. What they said.