3.5.2 Bias and variance of $\hat \mu$

Consider a random variable $\mathbf z\sim F_{\mathbf z}(\cdot )$. Let $\mu $ and $\sigma ^2$ the mean and the variance of $F_{\mathbf z}(\cdot )$, respectively. Suppose we have observed the i.i.d. sample $D_ N \leftarrow F_{\mathbf z}$. From (3.3.6), we obtain

\begin{equation} E_{{\mathbf D}_ N}[\hat{\boldsymbol{\mu }}]=E_{{\mathbf D}_ N}\left[\frac{1}{N} \sum _{i=1}^ N \mathbf z_ i \right]= \frac{\sum _{i=1}^ N E[\mathbf z_ i]}{N} =\frac{N \mu }{N}=\mu \tag{3.5.9} \label{eq-biasemu} \end{equation}

This means that the sample average estimator is not biased, whatever the distribution $F_{\mathbf z}(\cdot )$ is. And what about its variance? Since according to the i.i.d. assumption $\text {Cov}[\mathbf z_ i,\mathbf z_ j]=0$, for $i\neq j$, from (2.9.55) we obtain that the variance of the sample average estimator is

\begin{align} &\text {Var}\left[\hat{\boldsymbol {\mu}} \right] \tag{3.5.10} \label{eq-varemu} \\ &=\text {Var}\left[\frac{1}{N}\sum _{i=1}^ N \mathbf z_ i \right] \\ &= \frac{1}{N^2} \text {Var}\left[\sum _{i=1}^ N \mathbf z_ i \right] \\ &= \frac{1}{N^2} N \sigma ^2 \\ &=\frac{\sigma^2}{N}. \end{align}

In fact, $\hat{\boldsymbol {\mu }}$ is like the "round player" in dart game with some variance but no bias.

You can visualize the bias and variance of the sample average estimator by running the R script presented in Section 3.4.