Welcome back to Pattern Recognition.

So today we want to look a bit more into the independent component analysis and in particular

we want to think about what it actually means if the components are independent.

Now, let's build on the previous ideas.

So you can see that using the white-ending transform we were able to uncorrelate the

signals but the correlation is only determined by the second-degree cross moments, so the

covariances of a multivariate distribution.

Now, statistical independence is a much stronger statement than correlation as it determines

all of the cross moments.

So this then means that if we have two statistically independent variables y1 and y2 and two functions

h1 and h2 we can have a look at the expected value of the multiplication of the two.

So you could say h1 and h2 is the mixing and you could say y1 and y2 are the actual signal

sources.

Now, you can see I can write up this expected value as the double integral over h1 of y1

times h2 of y2 times the joint probability of y1 and y2 appearing together.

Now, we assume independence which means that we have the joint probability expressed as

the product of the individual probabilities and this then allows us to split the joint

probability and if we do so we can also split the integral into two integrals and now you

can see that the two remaining integrals only focus on one of the two variable which means

that I get the product of the expected values.

So the extra moment conditions allow us to identify the elements of A uniquely.

So we do more than the correlation.

In the case of a Gaussian distribution this is not the case because it is determined by

its second moments alone.

So in a Gaussian distribution you don't need the additional moments because if you have

the mean and the covariance matrix you have already determined the Gaussian distribution.

So any Gaussian independent components can be determined only up to a rotation.

Therefore we assume that the SI are independent and we assume that they are non-Gaussian because

if they are Gaussian our independent component analysis would simply not work.

If you look at the whitening transform then this is typically a pre-processing step of

our process.

So you would have this observation that the matrix A typically has n to the power of 2

degrees of freedom.

Now if you apply the whitening transform first you can actually see that we only need to

consider the transformed feature space.

So here X tilde is the transformed one and then we would apply our mixing matrix.

So we can essentially write this up as some A tilde and A tilde is essentially the matrix

that does the whitening transform and mixing together.

Now the new mixing matrix is orthogonal.

You can discover this if you look at the expected value of X tilde X tilde transpose.

So you look at the covariance matrix and we know our independent component analysis is

supposed to map onto our independent signals.

So we can pull out our A tilde and E tilde transpose on both sides and then you can see

that this expected value of the signal and the signal transpose is nothing else than

the covariant matrix of our signals.

They're independent so this is going to be the identity matrix and now you just have

A tilde times A tilde transpose and this has to map to the identity matrix and therefore

we can see that it's going to be an orthogonal matrix.

So A tilde is orthogonal and it has N times N minus one over two degrees of freedom and

thus applying the whitening transform already solves half of the problem.