2.10.1 Bivariate normal distribution

Let us consider a bivariate ($n=2$) normal density whose mean is $\mu =[\mu _1,\mu _2]^ T$ and the covariance matrix is

[ \Sigma = \begin{bmatrix} \sigma _{11} & \sigma _{12} \ \sigma _{21} & \sigma _{22} \end{bmatrix} ]

The correlation coefficient is

[ \rho =\frac{\sigma _{12}}{\sigma _1 \sigma _2} ]

It can be shown that the general bivariate normal density has the form

$$ p(z_1,z_2) =\frac{1}{2 \pi \sigma _1 \sigma _2 \sqrt {1-\rho ^2}} \exp \left[ -\frac{1}{2(1-\rho ^2)} \\ \left[ \left(\frac{z_1-\mu _1}{\sigma _1}\right)^2 -2 \rho \left( \frac{z_1-\mu _1}{\sigma _1} \right) \left( \frac{z_2-\mu _2}{\sigma _2} \right)+ \left(\frac{z_2-\mu _2}{\sigma _2}\right)^2 \right] \right] $$

A plot of a bivariate normal density with $\mu =[0,0]$ and $\Sigma =[1.2919, 0.4546;0.4546, 1.7081]$ and a corresponding contour curve are traced in Figure 2.11.

Bivariate normal density function Figure 2.11: Bivariate normal density function

R code
# script gaussXYZ.R
# It visualizes different bivariate gaussians with different
# orientation and axis' length

library(mvtnorm)
x <- seq(-10, 10, by= .5)
y <- x

z<-array(0,dim=c(length(x),length(y)))

#th : rotation angle of the first principal axis
#ax1: length principal axis 1
#ax2: length principal axis 2

ax1<-1

for (th in seq(0,pi,by=pi/8)){
  for (ax2 in c(4,12)){  
    Rot<-array(c(cos(th), -sin(th), sin(th), cos(th)),dim=c(2,2)); #rotation matrix
    A<-array(c(ax1, 0, 0, ax2),dim=c(2,2))
    Sigma<-(Rot%*%A)%*%t(Rot)
    E<-eigen(Sigma)
    print(paste("Eigenvalue of the Variance matrix=",E$values))
    print(E$vectors)
    for (i in 1:length(x)){
      for (j in 1:length(y)){
        z[i,j]<-dmvnorm(c(x[i],y[j]),sigma=Sigma)
      }
    }
    z[is.na(z)] <- 1
   
   
    op <- par(bg = "white")
    prob.z<-z
    par(ask=TRUE)
    persp(x, y, prob.z, theta = 30, phi = 30, expand = 0.5, col = "lightblue")
    #           contour(x,y,z)
    title (paste("BIVARIATE GAUSSIAN; Rot. angle=",
                 round(th,digits=3),"; Length axis 1=", ax1, "; Length axis 2=", ax2))
  }
}

One of the important properties of the multivariate normal density is that all conditional and marginal probabilities are also normal. Using the relation

[ p(z_2|z_1)=\frac{p(z_1,z_2)}{p(z_1)} ]

we find that $p(z_2|z_1)$ is a normal distribution $\mathcal{N}(\mu_{2|1}, \sigma^2_{2|1})$, where

\begin{align*} \mu _{2|1}& =\mu _2+\rho \frac{\sigma _2}{\sigma _1} (z_1-\mu _1)\\ \sigma ^2_{2|1}& =\sigma ^2_2(1-\rho ^2) \end{align*}

Note that

  • $\mu _{2|1}$ is a linear function of $z_1$: if the correlation coefficient $\rho $ is positive, the larger $z_1$, the larger $\mu _{2|1}$.

  • if there is no correlation between $z_1$ and $z_2$, the two variables are independent, i.e. we can ignore the value of $z_1$ to estimate $\mu _2$.