5.3.1 Random Samples

The probability distribution of any particular statistic depends not only on the population distribution (normal, uniform, etc.) and the sample size $n$ but also on the method of sampling. Consider selecting a sample of size $n=2$ from a population consisting of just the three values 1, 5, and 10, and suppose that the statistic of interest is the sample variance. If sampling is done “with replacement,” then $S^2=0$ will result if $X_1=X_2$. However, $S^2$ cannot equal 0 if sampling is “without replacement.” So $P\left(S^2=0\right)=0$ for one sampling method, and this probability is positive for the other method. Our next definition describes a sampling method often encountered (at least approximately) in practice.

random sample

The rv’s $X_1,X_2,\dots ,X_n$ are said to form a (simple) random sample of size $n$ if

1. The $X_i$’s are independent rv’s.
2. Every $X_i$ has the same probability distribution.

Conditions 1 and 2 can be paraphrased by saying that the $X_i$ ’s are independent and identically distributed (iid). If sampling is either with replacement or from an infinite (conceptual) population, Conditions 1 and 2 are satisfied exactly. These conditions will be approximately satisfied if sampling is without replacement, yet the sample size $n$ is much smaller than the population size $N$. In practice, if $n/N\le .05$ (at most $5\%$ of the population is sampled), we can proceed as if the $X_i$’s form a random sample. The virtue of such random sampling is that the probability distribution of any statistic can be more easily obtained than for any other sampling procedure.

There are two general methods for obtaining information about a statistic’s sampling distribution. One method involves calculations based on probability rules, and the other involves carrying out a simulation experiment.