3.5.1 Bias and variance

Once defined an estimator $\hat{\boldsymbol {\theta }}$ the next important thing is to assess how it is accurate. This leads us to the definition of bias, variance and standard error of an estimator.

Definition 5.33. (Bias of an estimator) An estimator $\hat{\boldsymbol {\theta }}$ of $\theta $ is said to be unbiased if and only if

$$ E_{{\mathbf D}_ N}[ \hat{\boldsymbol {\theta }}]=\theta $$

Otherwise, it is said to be biased with bias

\begin{equation} \text {Bias}[\hat{\boldsymbol {\theta}}]= E_{{\mathbf D}_ N}[\hat{\boldsymbol {\theta }}]-\theta \end{equation}

Definition 5.34. (Variance of an estimator) The variance of an estimator $\hat{\boldsymbol {\theta }}$ of $\theta $ is the variance of its sampling distribution

$$ \text{Var}\left[\hat{\boldsymbol {\theta }} \right]=E_{{\mathbf D}_N}[(\hat{\boldsymbol {\theta }}-E[\hat{\boldsymbol {\theta}}])^2] $$

Definition 5.35. (Standard error) The square root of the variance

[ \hat{\boldsymbol {\sigma }}=\sqrt {\text {Var}\left[\hat{\boldsymbol {\theta }} \right]} ]

is called the standard error of the estimator $\hat{\boldsymbol {\theta }}$.

Note that an unbiased estimator is an estimator that takes on average the right value. At the same time, many unbiased estimators may exist for a parameter $\theta $. Other important considerations are:

  • Given a generic transformation $f(\cdot )$, if $\hat{\boldsymbol {\theta }}$ is unbiased for $\theta $ does not imply that that $f(\hat{\boldsymbol {\theta }})$ is unbiased for $f(\theta )$.

  • A biased estimator with a known bias (not depending on $\theta $) is equivalent to an unbiased estimator since we can easily compensate for the bias.

  • In general the standard error $\hat{\boldsymbol {\sigma }}$ is not an unbiased estimator of $\sigma $ even if $\hat{\boldsymbol {\sigma }}^2$ is an unbiased estimator of $\sigma ^2$.

Example

An intuitive manner of visualizing the notion of sampling distribution of an estimator and the related concepts of bias and variance is to use the analogy of the dart game. The unknown parameter $\theta $ can be seen as the target and the estimator $\hat{\boldsymbol {\theta }}$ as a player. Figure 3.3 shows the target (black dot) together with the distribution of the draws of two different players: the C (cross) player and the R (round) player. In terms of our analogy the cross player/estimator has small variance but large bias, while the round one has small bias and large variance. Which is the best one?

The dart analogy Figure 3.3: The dart analogy: the target is the unknown parameter, the round dots represent some realizations of the estimator R, while the crosses represents some realizations of the estimator C.

$\bullet $

The following section will show that for a generic r.v. $\mathbf z$ and an i.i.d. dataset $D_ N$, the sample average $\hat{\boldsymbol {\mu }}$ and the sample variance $\hat{\boldsymbol {\sigma }}^2$ are unbiased estimators of the mean $E[\mathbf z]$ and the variance $\text {Var}\left[\mathbf z \right]$, respectively.