Welcome back to Pattern Recognition.

So today we want to look a bit more into the logistic regression and in particular we are

interested in how to estimate the actual regression parameters, so the decision boundaries.

So today we look more into logistic regression and until now we have seen that our F of x

was some arbitrary function and we've seen for example that it could be formulated as

a quadratic function as we've seen with the Gaussian distributions.

So generally we can express as a nonlinear function and the idea that we want to use

now is to linearize our estimation and what we will do is we will map our function into

a higher dimensional space.

So if you consider for example the quadratic function then we can see that we can express

it component wise in the following way.

So we know x is a vector consisting of x1 and x2, a can be written down in terms of

the individual components and the same is also true for alpha.

And if you look for the component wise notation you see that F of x can be written out in

components and we observe that the components that we have in there, they are linear.

So all the components of a and alpha are linear in this equation and the x's and so on they

appear in quadratic and linear terms.

So this means that we can rewrite this into a function of x with some x prime where x

prime is lifted to a six dimensional space.

So x prime is now rewritten from x1, x2 to x1 to the power of 2, x1, x2, x2 to the power

of 2, x1, x2 and 1.

And if we do that then we can rewrite the entire equation into an inner product of our

parameters and the parameter vector is now a11, a12 plus a21, a22, alpha1, alpha2 and

alpha0.

So this is a kind of interesting observation because it allows us to bring our nonlinear

quadratic function into a linear combination that is linear in the parameters that we wish

to estimate.

And this is a pretty cool observation because if we use this now we can essentially map

nonlinear functions into a higher dimensionality and in this higher dimensionality we are still

linear with respect to the parameters.

So we can now remember our logistic function and we can see that if we use this trick then

we can essentially take our logistic function and apply it in this high dimensional parameter

space and we don't have to touch the logistic function.

All that we have to do is we have to map essentially the x's into a higher dimensional space but

we can then use instead of the version where we were using the f of x which could be a

general nonlinear function we can replace it now with our parameter vector theta and

we essentially have the theta transpose x which is a linear decision boundary but in

a higher dimensional space and we can now use this and explore this idea a little further.

So also we want to assume the posteriores to be given by two classes so we have the

class y equals zero and y equals one.

If we do so we can write down the probabilities of the posteriores as one minus g of theta

transpose x and g theta transpose x where we essentially reuse our logistic function

or sigmoid function and the thing we are interested in now is the parameter vector theta.

So somehow we have to estimate theta from a set of m training observations and you remember

we are in the case of supervised learning here so we have some set s which is our training

data set and it contains m samples and they are essentially coupled observations where

we have some x1 y1 and more of these tuples up to m.

Now the method of choice is the maximum likelihood estimation.

If we want to use that then let's look a bit into the formulation how we want to write

the posteriores and we can rewrite this as a Bernoulli probability so the probability