6.2 Methods of Point Estimation

6.2 Methods of Point Estimation

We now introduce two “constructive” methods for obtaining point estimators: the method of moments and the method of maximum likelihood. By constructive we mean that the general definition of each type of estimator suggests explicitly how to obtain the estimator in any specific problem. Although maximum likelihood estimators are generally preferable to moment estimators because of certain efficiency properties, they often require significantly more computation than do moment estimators. It is sometimes the case that these methods yield unbiased estimators.

6.2.1 The Method of Moments

The basic idea of this method is to equate certain sample characteristics, such as the mean, to the corresponding population expected values. Then solving these equations for unknown parameter values yields the estimators.

DEFINITION Let $X_1,\dots ,X_n$ be a random sample from a pmf or $\mathrm{pdf}f\left(x\right)$ . For $k=1,2$ ,

$3,\dots$ , the $k$ th population moment, or $k$ th moment of the distribution $f\left(x\right)$ ,

is $E\left(X^k\right)$ . The $k$ th sample moment is $\left(1/n\right){\sum }_{i=1}^n{X}_i^k$ . Copyright 2016 Cengage Learning: All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has deemed that any suppressed content does not materially affect the overall learning experience. Congage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.

Thus the first population moment is $E\left(X\right)=\mu$ , and the first sample moment is $\sum X_i/n=\bar{X}$ . The second population and sample moments are $E\left(X^2\right)$ and $\sum X_i^2/n$ , respectively. The population moments will be functions of any unknown parameters ${\theta }_1,{\theta }_2,\dots$ .

DEFINITION Let $X_1,X_2,\dots ,X_n$ be a random sample from a distribution with pmf or pdf $f\left(x;{\theta }_1,\dots ,{\theta }_m\right)$ , where ${\theta }_1,\dots ,{\theta }_m$ are parameters whose values are unknown. Then the moment estimators ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_m$ are obtained by equating the first $m$ sample moments to the corresponding first $m$ population moments and solving for ${\theta }_1,\dots ,{\theta }_m$ .

If, for example, $m=2,E\left(X\right)$ and $E\left(X^2\right)$ will be functions of ${\theta }_1$ and ${\theta }_2$ . Setting $E\left(X\right)=\left(1/n\right)\sum X_i\left(=\bar{X}\right)$ and $E\left(X^2\right)=\left(1/n\right)\sum X_i^2$ gives two equations in ${\theta }_1$ and ${\theta }_2$ . The solution then defines the estimators.

EXAMPLE 6.12 Let $X_1,X_2,\dots ,X_n$ represent a random sample of service times of $n$ customers at a certain facility, where the underlying distribution is assumed exponential with parameter $\lambda$ . Since there is only one parameter to be estimated, the estimator is obtained by equating $E\left(X\right)$ to $\bar{X}$ . Since $E\left(X\right)=1/\lambda$ for an exponential distribution, this gives $1/\lambda =\bar{X}$ or $\lambda =1/\bar{X}$ . The moment estimator of $\lambda$ is then $\hat{\lambda }=1/\bar{X}$ .

AMPLE 6.13 Let $X_1,\dots ,X_n$ be a random sample from a gamma distribution with parameters $\alpha$ and $\beta$ . From Section 4.4, $E\left(X\right)=\alpha \beta$ and $E\left(X^2\right)={\beta }^2\Gamma \left(\alpha +2\right)/\Gamma \left(\alpha \right)={\beta }^2\left(\alpha +1\right)\alpha$ . The moment estimators of $\alpha$ and $\beta$ are obtained by solving

$$\bar{X}=\alpha \beta \frac{1}{n}\sum X_i^2=\alpha \left(\alpha +1\right){\beta }^2$$

Since $\alpha \left(\alpha +1\right){\beta }^2={\alpha }^2{\beta }^2+\alpha {\beta }^2$ and the first equation implies ${\alpha }^2{\beta }^2={\bar{X}}^2$ , the second equation becomes

$$\frac{1}{n}\sum X_i^2={\bar{X}}^2+\alpha {\beta }^2$$

Now dividing each side of this second equation by the corresponding side of the first equation and substituting back gives the estimators

$$\hat{\alpha }=\frac{{\bar{X}}^2}{\left(1/n\right)\sum X_i^2-{\bar{X}}^2}\hat{\beta }=\frac{\left(1/n\right)\sum X_i^2-{\bar{X}}^2}{\bar{X}}$$

To illustrate, the survival-time data mentioned in Example 4.24 is

$\begin{array}{llllllllll}152& 115& 109& 94& 88& 137& 152& 77& 160& 165\end{array}$

$\begin{array}{llllllllll}125& 40& 128& 123& 136& 101& 62& 153& 83& 69\end{array}$

from which $\bar{x}=113.5$ and $\left(1/20\right)\sum x_i^2=14,087.8$ . The parameter estimates are

$$\hat{\alpha }=\frac{{\left(113.5\right)}^2}{14,087.8-{\left(113.5\right)}^2}=10.7\hat{\beta }=\frac{14,087.8-{\left(113.5\right)}^2}{113.5}=10.6$$

These estimates of $\alpha$ and $\beta$ differ from the values suggested by Gross and Clark because they used a different estimation technique.

EXAMPLE 6.14 Let $X_1,\dots ,X_n$ be a random sample from a generalized negative binomial distribution with parameters $r$ and $p$ (see Section 3.5). Since $E\left(X\right)=r\left(1-p\right)/p$ and $V\left(X\right)=r\left(1-p\right)/p^2,E\left(X^2\right)=V\left(X\right)+{\left[E\left(X\right)\right]}^2=r\left(1-p\right)\left(r-rp+1\right)/p^2$ . Equating $E\left(X\right)$ to $\bar{X}$ and $E\left(X^2\right)$ to $\left(1/n\right)\sum X_i^2$ eventually gives

$$\hat{p}=\frac{\bar{X}}{\left(1/n\right)\sum X_i^2-{\bar{X}}^2}\hat{r}=\frac{{\bar{X}}^2}{\left(1/n\right)\sum X_i^2-{\bar{X}}^2-\bar{X}}$$

As an illustration, Reep, Pollard, and Benjamin (“Skill and Chance in Ball Games,” J. of Royal Stat. Soc., 1971: 623-629) consider the negative binomial distribution as a model for the number of goals per game scored by National Hockey League teams. The data for 1966-1967 follows (420 games):

Goals	0	1	2	3	4	5	6	7	8	9	10
Frequency	29	71	82	89	65	45	24	7	4	1	3

Then,

$$\bar{x}=\sum x_i/420=\left[\left(0\right)\left(29\right)+\left(1\right)\left(71\right)+\cdots +\left(10\right)\left(3\right)\right]/420=2.98$$

and

$$\sum x_i^2/420=\left[{\left(0\right)}^2\left(29\right)+{\left(1\right)}^2\left(71\right)+\cdots +{\left(10\right)}^2\left(3\right)\right]/420=12.40$$

Thus,

$$\hat{p}=\frac{2.98}{12.40-{\left(2.98\right)}^2}=.85\hat{r}=\frac{{\left(2.98\right)}^2}{12.40-{\left(2.98\right)}^2-2.98}=16.5$$

Although $r$ by definition must be positive, the denominator of $\hat{r}$ could be negative, indicating that the negative binomial distribution is not appropriate (or that the moment estimator is flawed).

6.2.2 Maximum Likelihood Estimation

The method of maximum likelihood was first introduced by R. A. Fisher, a geneticist and statistician, in the 1920s. Most statisticians recommend this method, at least when the sample size is large, since the resulting estimators have certain desirable efficiency properties (see the proposition on page 271).

XAMPLE 6.15 The best protection against hacking into an online account is to use a password that has at least 8 characters consisting of upper- and lowercase letters, numerals, and special characters. [Note: The Jan. 2012 issue of Consumer Reports reported that only $25\%$ of individuals surveyed used a strong password.] Suppose that 10 individuals who have email accounts with a certain provider are selected, and it is found that the first, third, and tenth individuals have such strong protection, whereas the others do not. Let $p=P$ (strong protection), i.e., $p$ is the proportion of all such account holders having strong protection. Define (Bernoulli) random variables $X_1,X_2,\dots ,X_{10}$ by

$$X_1=\{\begin{array}{l}1\text{ if }1\text{ st has strong protection }\\ 0\text{ if }1\text{ st does not have strong protection }\end{array}\dots X_{10}=\{\begin{array}{l}1\text{ if }10\text{ th has strong protection }\\ 0\text{ if }10\text{ th does not have strong protection }\end{array}$$

Then for the obtained sample, $X_1=X_3=X_{10}=1$ and the other seven $X_i$ ’s are all zero. The probability mass function of any particular $X_i$ is $p^{x_i}{\left(1-p\right)}^{1-x_i}$ , which becomes $p$ if $x_i=1$ and $1-p$ when $x_i=0$ . Now suppose that the conditions of various passwords are independent of one another. This implies that the $X_i$ ’s are independent, so their joint probability mass function is the product of the individual pmf’s. Thus the joint pmf evaluated at the observed $X_i$ ’s is

$$f\left(x_1,\dots ,x_{10};p\right)=p\left(1-p\right)p\cdots p=p^3{\left(1-p\right)}^7\text{\hspace{1em}(6.4)}$$

Suppose that $p=.25$ . Then the probability of observing the sample that we actually obtained is ${\left(.25\right)}^3{\left(.75\right)}^7=.002086$ . If instead $p=.50$ , then this probability is ${\left(.50\right)}^3{\left(.50\right)}^7=.000977$ . For what value of $p$ is the obtained sample most likely to have occurred? That is, for what value of $p$ is the joint pmf (6.4) as large as it can be? What value of $p$ maximizes (6.4)? Figure 6.6(a) shows a graph of the likelihood (6.4) as a function of $p$ . It appears that the graph reaches its peak above $p=.3=$ the proportion of flawed helmets in the sample. Figure 6.6(b) shows a graph of the natural logarithm of (6.4); since $\ln \left[\mathrm{g}\left(u\right)\right]$ is a strictly increasing function of $g\left(u\right)$ , finding $u$ to maximize the function $g\left(u\right)$ is the same as finding $u$ to maximize $\ln \left[\mathrm{g}\left(u\right)\right]$ .

Figure 6.6 (a) Graph of the likelihood (joint pmf) (6.4) from Example 6.15 (b) Graph of the natural logarithm of the likelihood

We can verify our visual impression by using calculus to find the value of $p$ that maximizes (6.4). Working with the natural $\log$ of the joint pmf is often easier than working with the joint pmf itself, since the joint pmf is typically a product so its logarithm will be a sum. Here

$$\ln \left[f\left(x_1,\dots ,x_{10};p\right)\right]=\ln \left[p^3{\left(1-p\right)}^7\right]=3\ln \left(p\right)+7\ln \left(1-p\right)\text{\hspace{1em}(6.5)}$$

Thus

$$\frac{d}{dp}\left\{\ln \left[f\left(x_1,\dots ,x_{10};p\right)\right]\right\}=\frac{d}{dp}\{3\ln \left(p\right)+7\ln \left(1-p\right)\}=\frac{3}{p}+\frac{7}{1-p}\left(-1\right)$$ $$=\frac{3}{p}-\frac{7}{1-p}$$

[the $\left(-1\right)$ comes from the chain rule in calculus]. Equating this derivative to 0 and solving for $p$ gives $3\left(1-p\right)=7p$ , from which $3=10p$ and so $p=3/10=.30$ as conjectured. That is, our point estimate is $\hat{p}=.30$ . It is called the maximum likelihood estimate because it is the parameter value that maximizes the likelihood (joint pmf) of the observed sample. In general, the second derivative should be examined to make sure a maximum has been obtained, but here this is obvious from Figure 6.5.

Suppose that rather than being told the condition of every password, we had only been informed that three of the ten were strong. Then we would have the observed value of a binomial random variable $X=$ the number with strong passwords. The pmf of $X$ is $\left(\begin{array}{c}10\\ x\end{array}\right)p^x{\left(1-p\right)}^{10-x}$ . For $x=3$ , this becomes $\left(\begin{array}{c}10\\ 3\end{array}\right)p^3{\left(1-p\right)}^7$ . The binomial coefficient $\left(\begin{array}{c}10\\ 3\end{array}\right)$ is irrelevant to the maximization, so again $\hat{p}=.30$ .

---

DEFINITION

---

Let $X_1,X_2,\dots ,X_n$ have joint pmf or pdf

$$f\left(x_1,x_2,\dots ,x_n;{\theta }_1,\dots ,{\theta }_m\right)\text{\hspace{1em}(6.6)}$$

where the parameters ${\theta }_1,\dots ,{\theta }_m$ have unknown values. When $x_1,\dots ,x_n$ are the observed sample values and (6.6) is regarded as a function of ${\theta }_1,\dots ,{\theta }_m$ , it is called the likelihood function. The maximum likelihood estimates (mle’s) ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_m$ are those values of the ${\theta }_i$ ’s that maximize the likelihood function, so that

$$f\left(x_1,\dots ,x_n;{\hat{\theta }}_1,\dots ,{\hat{\theta }}_m\right)\ge f\left(x_1,\dots ,x_n;{\theta }_1,\dots ,{\theta }_m\right)\text{for all}{\theta }_1,\dots ,{\theta }_m$$

When the $X_i$ ’s are substituted in place of the $x_i$ ’s, the maximum likelihood estimators result.

The likelihood function tells us how likely the observed sample is as a function of the possible parameter values. Maximizing the likelihood gives the parameter values for which the observed sample is most likely to have been generated-that is, the parameter values that “agree most closely” with the observed data.

EXAMPLE 6.16 Suppose $X_1,X_2,\dots ,X_n$ is a random sample from an exponential distribution with parameter $\lambda$ . Because of independence, the likelihood function is a product of the individual pdf’s:

$$f\left(x_1,\dots ,x_n;\lambda \right)=\left(\lambda e^{-\lambda x_1}\right)\cdots \cdots \left(\lambda e^{-\lambda x_n}\right)={\lambda }^n{e}^{-\lambda \sum x_i}$$

The natural logarithm of the likelihood function is

$$\ln \left[f\left(x_1,\dots ,x_n;\lambda \right)\right]=n\ln \left(\lambda \right)-\lambda \sum x_i$$

Equating $\left(d/d\lambda \right)\left[\ln \left(\text{likelihood}\right)\right]$ to zero results in $n/\lambda -\sum x_i=0$ , or $\lambda =n/\sum x_i=1/\bar{x}$ . Thus the mle is $\hat{\lambda }=1/\bar{X}$ ; it is identical to the method of moments estimator [but it is not an unbiased estimator, since $E\left(1/\bar{X}\right)\ne 1/E\left(\bar{X}\right)]$ .

EXAMPLE 6.17 Let $X_1,\dots ,X_n$ be a random sample from a normal distribution. The likelihood function is

$$f\left(x_1,\dots ,x_n;\mu ,{\sigma }^2\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-{\left(x_1-\mu \right)}^2/\left(2{\sigma }^2\right)}\cdot \cdots \cdot \frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-{\left(x_n-\mu \right)}^2/\left(2{\sigma }^2\right)}$$ $$={\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}{e}^{-\sum {\left(x_i-\mu \right)}^2/\left(2{\sigma }^2\right)}$$

so

$$\ln \left[f\left(x_1,\dots ,x_n;\mu ,{\sigma }^2\right)\right]=-\frac{n}{2}\ln \left(2\pi {\sigma }^2\right)-\frac{1}{2{\sigma }^2}\sum {\left(x_i-\mu \right)}^2$$

To find the maximizing values of $\mu$ and ${\sigma }^2$ , we must take the partial derivatives of $\ln \left(f\right)$ with respect to $\mu$ and ${\sigma }^2$ , equate them to zero, and solve the resulting two equations. Omitting the details, the resulting mle’s are

$$\hat{\mu }=\bar{X}{\hat{\sigma }}^2=\frac{\sum {\left(X_i-\bar{X}\right)}^2}{n}$$

The mle of ${\sigma }^2$ is not the unbiased estimator, so two different principles of estimation (unbiasedness and maximum likelihood) yield two different estimators.

EXAMPLE 6.18 In Chapter 3, we mentioned the use of the Poisson distribution for modeling the number of “events” that occur in a two-dimensional region. Assume that when the region $R$ being sampled has area $a\left(R\right)$ , the number $X$ of events occurring in $R$ has a Poisson distribution with parameter $\lambda a\left(R\right)$ (where $\lambda$ is the expected number of events per unit area) and that nonoverlapping regions yield independent $X$ ’s.

Suppose an ecologist selects $n$ nonoverlapping regions $R_1,\dots ,R_n$ and counts the number of plants of a certain species found in each region. The joint pmf (likelihood) is then

$$p\left(x_1,\dots ,x_n;\lambda \right)=\frac{{\left[\lambda \cdot a\left(R_1\right)\right]}^{x_1}{e}^{-\lambda \cdot a\left(R_1\right)}}{x_1!}\cdot \dots \cdot \frac{{\left[\lambda \cdot a\left(R_n\right)\right]}^{x_n}{e}^{-\lambda \cdot a\left(R_n\right)}}{x_n!}$$ $$=\frac{{\left[a\left(R_1\right)\right]}^{x_1}\cdot \cdots \cdot {\left[a\left(R_n\right)\right]}^{x_n}\cdot {\lambda }^{\sum x_i}\cdot e^{-\lambda \sum a\left(R_i\right)}}{x_1!\cdot \cdots \cdot x_n!}$$

The log likelihood is

$$\ln \left[p\left(x_1,\dots ,x_n;\lambda \right)\right]=\sum x_i\cdot \ln \left[a\left(R_i\right)\right]+\ln \left(\lambda \right)\cdot \sum x_i-\lambda \sum a\left(R_i\right)-\sum \ln \left(x_i!\right)$$

Taking $d/d\lambda \left[\ln \left(p\right)\right]$ and equating it to zero yields

$$\frac{\sum x_i}{\lambda }-\sum a\left(R_i\right)=0$$

from which

$$\lambda =\frac{\sum x_i}{\sum a\left(R_i\right)}$$

The mle is then $\hat{\lambda }=\sum X_i/\sum a\left(R_i\right)$ . This is intuitively reasonable because $\lambda$ is the true density (plants per unit area), whereas $\hat{\lambda }$ is the sample density since $\sum a\left(R_i\right)$ is just the total area sampled. Because $E\left(X_i\right)=\lambda \cdot a\left(R_i\right)$ , the estimator is unbiased.

Sometimes an alternative sampling procedure is used. Instead of fixing regions to be sampled, the ecologist will select $n$ points in the entire region of interest and let $y_i=$ the distance from the $i$ th point to the nearest plant. The cumulative distribution function (cdf) of $Y=$ distance to the nearest plant is

$$F_Y\left(y\right)=P\left(Y\le y\right)=1-P\left(Y>y\right)=1-P\left(\begin{array}{c}\text{ no plants in }\mathrm{a}\\ \text{ circle of radius }y\end{array}\right)$$ $$=1-\frac{e^{-\lambda \pi y^2}{\left(\lambda \pi y^2\right)}^0}{0!}=1-e^{-\lambda \cdot \pi y^2}$$

Taking the derivative of $F_Y\left(y\right)$ with respect to $y$ yields

$$f_Y\left(y;\lambda \right)=\{\begin{array}{cc}2\pi \lambda y{e}^{-\lambda \pi y^2}& y\ge 0\\ 0& \text{ otherwise }\end{array}$$

If we now form the likelihood $f_Y\left(y_1;\lambda \right)\cdot \dots \cdot f_Y\left(y_n;\lambda \right)$ , differentiate ln(likelihood), and so on, the resulting mle is

$$\hat{\lambda }=\frac{n}{\pi \sum Y_i^2}=\frac{\text{ number of plants observed }}{\text{ total area sampled }}$$

which is also a sample density. It can be shown that in a sparse environment (small $\lambda$ ), the distance method is in a certain sense better, whereas in a dense environment the first sampling method is better.

EXAMPLE 6.19 Let $X_1,\dots ,X_n$ be a random sample from a Weibull pdf

$$f\left(x;\alpha ,\beta \right)=\{\begin{array}{cc}\frac{\alpha }{{\beta }^{\alpha }}\cdot x^{\alpha -1}\cdot e^{-{\left(x/\beta \right)}^{\alpha }}& x\ge 0\\ 0& \text{ otherwise }\end{array}$$

Writing the likelihood and ln(likelihood), then setting both $\left(\partial /\partial \alpha \right)\left[\ln \left(f\right)\right]=0$ and $\left(\partial /\partial \beta \right)\left[\ln \left(f\right)\right]=0$ , yields the equations

$$\alpha ={\left[\frac{\sum x_i^{\alpha }\cdot \ln \left(x_i\right)}{\sum x_i^{\alpha }}-\frac{\sum \ln \left(x_i\right)}{n}\right]}^{-1}\beta ={\left(\frac{\sum x_i^{\alpha }}{n}\right)}^{1/\alpha }$$

These two equations cannot be solved explicitly to give general formulas for the mle’s $\hat{\alpha }$ and $\hat{\beta }$ . Instead, for each sample $x_1,\dots ,x_n$ , the equations must be solved using an iterative numerical procedure. The R, SAS and Minitab software packages can be used for this purpose. Even moment estimators of $\alpha$ and $\beta$ are somewhat complicated (see Exercise 21).

6.2.3 Estimating Functions of Parameters

Once the mle for a parameter $\theta$ is available, the mle for any function of $\theta$ , such as $1/\theta$ or $\sqrt{\theta }$ , is easily obtained.

---

PROPOSITION

---

The Invariance Principle

Let ${\hat{\theta }}_1,{\hat{\theta }}_2,\dots ,{\hat{\theta }}_m$ be the mle’s of the parameters ${\theta }_1,{\theta }_2,\dots ,{\theta }_m$ . Then the mle of any function $h\left({\hat{\theta }}_1,{\theta }_2,\dots ,{\theta }_m\right)$ of these parameters is the function $h\left({\hat{\theta }}_1,{\hat{\theta }}_2,\dots ,{\hat{\theta }}_m\right)$ of the mle’s.

---

continued) function:

---

In the normal case, the mle’s of $\mu$ and ${\sigma }^2$ are $\hat{\mu }=\bar{X}$ and ${\hat{\sigma }}^2=\sum {\left(X_i-\bar{X}\right)}^2/n$ . To obtain the mle of the function $h\left(\mu ,{\sigma }^2\right)=\sqrt{{\sigma }^2}=\sigma$ , substitute the mle’s into the

$$\hat{\sigma }=\sqrt{{\hat{\sigma }}^2}={\left[\frac{1}{n}\sum {\left(X_i-\bar{X}\right)}^2\right]}^{1/2}$$

The mle of $\sigma$ is not the sample standard deviation $S$ , though they are close unless $n$ is quite small.

EXAMPLE 6.21 The mean value of an rv $X$ that has a Weibull distribution is

(Example 6.19

continued)

$$\mu =\beta \cdot \Gamma \left(1+1/\alpha \right)$$

The mle of $\mu$ is therefore $\hat{\mu }=\hat{\beta }\Gamma \left(1+1/\hat{\alpha }\right)$ , where $\hat{\alpha }$ and $\hat{\beta }$ are the mle’s of $\alpha$ and $\beta$ . In particular, $\bar{X}$ is not the mle of $\mu$ , though it is an unbiased estimator. At least for large $n,\hat{\mu }$ is a better estimator than $\bar{X}$ .

For the data given in Example 6.3, the mle’s of the Weibull parameters are $\hat{\alpha }=11.9731$ and $\hat{\beta }=77.0153$ , from which $\hat{\mu }=73.80$ . This estimate is quite close to the sample mean 73.88.

6.2.4 Large Sample Behavior of the MLE

Although the principle of maximum likelihood estimation has considerable intuitive appeal, the following proposition provides additional rationale for the use of mle’s.

---

PROPOSITION

---

Under very general conditions on the joint distribution of the sample, when the sample size $n$ is large, the maximum likelihood estimator of any parameter $\theta$ is at least approximately unbiased $\left[E\left(\hat{\theta }\right)\approx \theta \right]$ and has variance that is either as small as or nearly as small as can be achieved by any estimator. Stated another way, the mle $\hat{\theta }$ is either exactly or at least approximately the MVUE of $\theta$ .

Because of this result and the fact that calculus-based techniques can usually be used to derive the mle’s (though often numerical methods, such as Newton’s method, are necessary), maximum likelihood estimation is the most widely used estimation technique among statisticians. Many of the estimators used in the remainder of the book are mle’s. Obtaining an mle, however, does require that the underlying distribution be specified.

6.2.5 Some Complications

Sometimes calculus cannot be used to obtain mle’s.

---

EXAMPLE 6.22

---

2 Suppose waiting time for a bus is uniformly distributed on $\left[0,\theta \right]$ and the results $x_1,\dots ,x_n$ of a random sample from this distribution have been observed. Since $f\left(x;\theta \right)=1/\theta$ for $0\le x\le \theta$ and 0 otherwise,

$$f\left(x_1,\dots ,x_n;\theta \right)=\{\begin{array}{ll}\frac{1}{{\theta }^n}& 0\le x_1\le \theta ,\dots ,0\le x_n\le \theta \\ 0& \text{ otherwise }\end{array}$$

As long as $\max \left(x_i\right)\le \theta$ , the likelihood is $1/{\theta }^n$ , which is positive, but as soon as $\theta <\max \left(x_i\right)$ , the likelihood drops to 0 . This is illustrated in Figure 6.7. Calculus will not work because the maximum of the likelihood occurs at a point of discontinuity, but the figure shows that $\hat{\theta }=\max \left(X_i\right)$ . Thus if my waiting times are $2.3,3.7,1.5,.4$ , and 3.2, then the mle is $\hat{\theta }=3.7$ . From Example 6.4, the mle is not unbiased. Copyright 2016 Congage Learning. All Rights Reserved, May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/oreChapter(s). Editorial review has deemed that any suppressed content does not materially affect the overall learning experience. Congage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.

Figure 6.7 The likelihood function for Example 6.22

EXAMPLE 6.23 A method that is often used to estimate the size of a wildlife population involves performing a capture/recapture experiment. In this experiment, an initial sample of $M$ animals is captured, each of these animals is tagged, and the animals are then returned to the population. After allowing enough time for the tagged individuals to mix into the population, another sample of size $n$ is captured. With $X=$ the number of tagged animals in the second sample, the objective is to use the observed $x$ to estimate the population size $N$ .

The parameter of interest is $\theta =N$ , which can assume only integer values, so even after determining the likelihood function (pmf of $X$ here), using calculus to obtain $N$ would present difficulties. If we think of a success as a previously tagged animal being recaptured, then sampling is without replacement from a population containing $M$ successes and $N-M$ failures, so that $X$ is a hypergeometric rv and the likelihood function is

$$p\left(x;N\right)=h\left(x;n,M,N\right)=\frac{\left(\begin{array}{c}M\\ x\end{array}\right)\cdot \left(\begin{array}{c}N-M\\ n-x\end{array}\right)}{\left(\begin{array}{l}N\\ n\end{array}\right)}$$

The integer-valued nature of $N$ notwithstanding, it would be difficult to take the derivative of $p\left(x;N\right)$ . However, if we consider the ratio of $p\left(x;N\right)$ to $p\left(x;N-1\right)$ , we have

$$\frac{p\left(x;N\right)}{p\left(x;N-1\right)}=\frac{\left(N-M\right)\cdot \left(N-n\right)}{N\left(N-M-n+x\right)}$$

This ratio is larger than 1 if and only if (iff) $N<Mn/x$ . The value of $N$ for which $p\left(x;N\right)$ is maximized is therefore the largest integer less than $Mn/x$ . If we use standard mathematical notation $\left[r\right]$ for the largest integer less than or equal to $r$ , the mle of $N$ is $\hat{N}=\left[Mn/x\right]$ . As an illustration, if $M=200$ fish are taken from a lake and tagged, and subsequently $n=100$ fish are recaptured, and among the 100 there are $x=11$ tagged fish, then $\hat{N}=\left[\left(200\right)\left(100\right)/11\right]=\left[1818.18\right]=1818$ . The estimate is actually rather intuitive; $x/n$ is the proportion of the recaptured sample that is tagged, whereas $M/N$ is the proportion of the entire population that is tagged. The estimate is obtained by equating these two proportions (estimating a population proportion by a sample proportion).

Suppose $X_1,X_2,\dots ,X_n$ is a random sample from a pdf $f\left(x;\theta \right)$ that is symmetric about $\theta$ but that the investigator is unsure of the form of the $f$ function. It is then desirable to use an estimator $\hat{\theta }$ that is robust-that is, one that performs well for a wide variety of underlying pdf’s. One such estimator is a trimmed mean. In recent years, statisticians have proposed another type of estimator, called an $M$ -estimator; based on a generalization of maximum likelihood estimation. Instead of maximizing the log likelihood $\sum \ln \left[f\left(x;\theta \right)\right]$ for a specified $f$ , one maximizes $\sum \rho \left(x_i;\theta \right)$ . The “objective function” $\rho$ is selected to yield an estimator with good robustness properties. The book by David Hoaglin et al. (see the bibliography) contains a good exposition of this topic.

EXERCISES Section 6.2 (20–30)

20. A diagnostic test for a certain disease is applied to $n$ individuals known to not have the disease. Let $X=$ the number among the $n$ test results that are positive (indicating presence of the disease, so $X$ is the number of false positives) and $p=$ the probability that a disease-free individual’s test result is positive (i.e., $p$ is the true proportion of test results from disease-free individuals that are positive). Assume that only $X$ is available rather than the actual sequence of test results.

a. Derive the maximum likelihood estimator of $p$ . If $n=20$ and $x=3$ , what is the estimate?

b. Is the estimator of part (a) unbiased?

c. If $n=20$ and $x=3$ , what is the mle of the probability ${\left(1-p\right)}^5$ that none of the next five tests done on disease-free individuals are positive?

21. Let $X$ have a Weibull distribution with parameters $\alpha$ and $\beta$ , so

$$E\left(X\right)=\beta \cdot \Gamma \left(1+1/\alpha \right)$$ $$V\left(X\right)={\beta }^2\left\{\Gamma \left(1+2/\alpha \right)-{\left[\Gamma \left(1+1/\alpha \right)\right]}^2\right\}$$

a. Based on a random sample $X_1,\dots ,X_n$ , write equations for the method of moments estimators of $\beta$ and $\alpha$ . Show that, once the estimate of $\alpha$ has been obtained, the estimate of $\beta$ can be found from a table of the gamma function and that the estimate of $\alpha$ is the solution to a complicated equation involving the gamma function.

b. If $n=20,\bar{x}=28.0$ , and $\sum x_i^2=16,500$ , compute the estimates. [Hint: ${\left[\Gamma \left(1.2\right)\right]}^2/\Gamma \left(1.4\right)=.95$ .]

22. Let $X$ denote the proportion of allotted time that a randomly selected student spends working on a certain aptitude test. Suppose the pdf of $X$ is

$$f\left(x;\theta \right)=\{\begin{array}{cc}\left(\theta +1\right)x^{\theta }& 0\le x\le 1\\ 0& \text{ otherwise }\end{array}$$

where $-1<\theta$ . A random sample of ten students yields data $x_1=.92,x_2=.79,x_3=.90,x_4=.65,x_5=.86$ , $x_6=.47,x_7=.73,x_8=.97,x_9=.94,x_{10}=.77$ .

a. Use the method of moments to obtain an estimator of $\theta$ , and then compute the estimate for this data.

b. Obtain the maximum likelihood estimator of $\theta$ , and then compute the estimate for the given data.

23. Let $X$ represent the error in making a measurement of a physical characteristic or property (e.g., the boiling point of a particular liquid). It is often reasonable to assume that $E\left(X\right)=0$ and that $X$ has a normal distribution. Thus the pdf of any particular measurement error is

$$f\left(x;\theta \right)=\frac{1}{\sqrt{2\pi \theta }}{e}^{-x^2/2\theta }-\infty <x<\infty$$

(where we have used $\theta$ in place of ${\sigma }^2$ ). Now suppose that $n$ independent measurements are made, resulting in measurement errors $X_1=x_1,X_2=x_2,\dots ,X_n=x_n$ . Obtain the mle of $\theta$ .

24. A vehicle with a particular defect in its emission control system is taken to a succession of randomly selected mechanics until $r=3$ of them have correctly diagnosed the problem. Suppose that this requires diagnoses by 20 different mechanics (so there were 17 incorrect diagnoses). Let $p=P$ (correct diagnosis), so $p$ is the proportion of all mechanics who would correctly diagnose the problem. What is the mle of $p$ ? Is it the same as the mle if a random sample of 20 mechanics results in 3 correct diagnoses? Explain. How does the mle compare to the estimate resulting from the use of the unbiased estimator given in Exercise 17?

25. The shear strength of each of ten test spot welds is determined, yielding the following data (psi):

$\begin{array}{llllllllll}392& 376& 401& 367& 389& 362& 409& 415& 358& 375\end{array}$

a. Assuming that shear strength is normally distributed, estimate the true average shear strength and standard deviation of shear strength using the method of maximum likelihood.

b. Again assuming a normal distribution, estimate the strength value below which $95\%$ of all welds will have their strengths. [Hint: What is the 95th percentile in terms of $\mu$ and $\sigma$ ? Now use the invariance principle.]

c. Suppose we decide to examine another test spot weld. Let $X=$ shear strength of the weld. Use the given data to obtain the mle of $P\left(X\le 400\right)$ . [Hint: $P\left(X\le 400\right)=\Phi \left(\left(400-\mu \right)/\sigma \right).]$

26. Consider randomly selecting $n$ segments of pipe and determining the corrosion loss (mm) in the wall thickness for each one. Denote these corrosion losses by $Y_1,\dots ,Y_n$ . The article “A Probabilistic Model for a Gas Explosion Due to Leakages in the Grey Cast Iron Gas Mains” (Reliability Engr. and System Safety (2013:270-279) proposes a linear corrosion model: $Y_i=t_{i}R$ , where $t_i$ is the age of the pipe and $R$ , the corrosion rate, is exponentially distributed with parameter $\lambda$ . Obtain the maximum likelihood estimator of the exponential parameter (the resulting mle appears in the cited article). [Hint: If $c>0$ and $X$ has an exponential distribution, so does $cX$ .]

27. Let $X_1,\dots ,X_n$ be a random sample from a gamma distribution with parameters $\alpha$ and $\beta$ .

a. Derive the equations whose solutions yield the maximum likelihood estimators of $\alpha$ and $\beta$ . Do you think they can be solved explicitly?

b. Show that the mle of $\mu =\alpha \beta$ is $\hat{\mu }=\bar{X}$ .

28. Let $X_1,X_2,\dots ,X_n$ represent a random sample from the Rayleigh distribution with density function given in Exercise 15. Determine

a. The maximum likelihood estimator of $\theta$ , and then calculate the estimate for the vibratory stress data given in that exercise. Is this estimator the same as the unbiased estimator suggested in Exercise 15?

b. The mle of the median of the vibratory stress distribution. [Hint: First express the median in terms of $\theta$ .]

29. Consider a random sample $X_1,X_2,\dots ,X_n$ from the shifted exponential pdf

$$f\left(x;\lambda ,\theta \right)=\{\begin{array}{cc}\lambda e^{-\lambda \left(x-\theta \right)}& x\ge \theta \\ 0& \text{ otherwise }\end{array}$$

Taking $\theta =0$ gives the pdf of the exponential distribution considered previously (with positive density to the right of zero). An example of the shifted exponential distribution appeared in Example 4.5, in which the variable of interest was time headway in traffic flow and $\theta =.5$ was the minimum possible time headway.

a. Obtain the maximum likelihood estimators of $\theta$ and $\lambda$ .

b. If $n=10$ time headway observations are made, resulting in the values $3.11,.64,2.55,2.20,5.44$ , $3.42,10.39,8.93,17.82$ , and 1.30, calculate the estimates of $\theta$ and $\lambda$ .

30. At time $t=0,20$ identical components are tested. The lifetime distribution of each is exponential with parameter $\lambda$ . The experimenter then leaves the test facility unmonitored. On his return 24 hours later, the experimenter immediately terminates the test after noticing that $y=15$ of the 20 components are still in operation (so 5 have failed). Derive the mle of $\lambda$ . [Hint: Let $Y=$ the number that survive 24 hours. Then $Y\sim \mathrm{Bin}\left(n,p\right)$ . What is the mle of $p$ ? Now notice that $p=P\left(X_i\ge 24\right)$ , where $X_i$ is exponentially distributed. This relates $\lambda$ to $p$ , so the former can be estimated once the latter has been.]