aka mixture of Gaussians

Example in 1D

Algo def

Assume there's a latent (hidden/unobserved) Random variable $z$ and $x^{(i)}, z^{(i)}$ have a joint distribution

$P(x^{(i)}, z^{(i)})=P(x^{(i)}|z^{(i)})P(z^{(i)})$

$z^{(i)} \sim \text{Multinomial}(\phi)$

($\phi_j \geq 0 \sum_j \phi_j =1$)

$x^{(i)} | (z^{(i)} = j) \sim \mathcal{N}(\mu_j, \Sigma_j)$

This is very similar to Gaussian discriminant analysis, but where the known labels are substituted by unknown hidden variables $z$. (vid).

Train via the EM algorithm

See video – EM for mixture of Gaussians

1. Repeat until convergence
   1. E-step. Guess values of $z^{(i)}$s. In particular, compute the a-posteriori probability $w^{(i)}_j = P(z^{(i)} = j | x^{(i)}; \phi, \mu, \Sigma)$ $= \frac{P(x^{(i)}|z^{(i)} = j)P(z^{(i)} = j)}{\sum_{l=1}^k P(x^{(i)}|z^{(i)} = l)P(z^{(i)} = l)}$ $=\frac{\frac{1}{\left(2\pi \right)^{\frac{d}{2}}\left|\Sigma _j\right|^{\frac{1}{2}}}\exp \left\{\left(x^{\left(i\right)}-\mu _j\right)^T\Sigma _j^{-1}\left(x^{\left(i\right)}-\mu _j\right)\right\}\phi _j}{\sum _{l=1}^k\frac{1}{\left(2\pi \right)^{\frac{d}{2}}\left|\Sigma _l\right|^{\frac{1}{2}}}\exp \left\{\left(x^{\left(i\right)}-\mu _l\right)^T\Sigma _l^{-1}\left(x^{\left(i\right)}-\mu _l\right)\right\}\phi _l}$
   2. M-step. $\phi _j=\frac{1}{m}\sum _{_{i=1}}^mw_j^{\left(i\right)}$. $\mu _j=\frac{\sum _{_{i=1}}^mw_j^{\left(i\right)}x^{\left(i\right)}}{\sum _{_{i=1}}^mw_j^{\left(i\right)}}$. $\Sigma _j=\frac{\sum _{i=1}^mw_j^{\left(i\right)}\left(x^{\left(i\right)}-\mu _j\right)\left(x^{\left(i\right)}-\mu _j\right)^T}{\sum _{i=1}^mw_j^{\left(i\right)}}$