Question number three is about minimizing phi of x where phi of x is 1 half x

transpose bx minus b transpose x and b has the shape 2 minus 1 minus 1 2.

Lowcase b has form 3 and 0. First part of the question is find the analytical, find

the minimizer analytically and we know that in order to have this we have to

put the gradient to 0. Gradient is bx minus b this has to be equal to 0 so we

have to solve the system 2 minus 1 minus 1 2 times x1 x2 is equal to 3 and 0.

Okay we can do this 2x1 minus x2 is equal to 3 and minus x1 plus 2x2 is equal to 0.

And this is very elementary computation let's say 1 plus 2 times the second row this

gives us 0 times x1 plus 3x2 is equal to 3 so this means that x2 is equal to 1

and put this in let's say a second line then you see that x1 is equal to 2x2

which is equal to 2 so the minimizer is 2 1 this is the minimizer let's call this

x star. Okay and the second part, part b is to try and find this minimizer with an

iterative optimization method with gradient descent this is just running

this algorithm we start with 0 0 as the initial starting point then r0 the first

residual is b lowercase b minus uppercase b times x0 which is equal to 3 0 and

well what's alpha 0 the optimal step size this is 3 0 trans not not transpose

this is already transposed times 3 0 divided by 3 0 times 2 minus 1 minus 1 2 times 3 0

which is 9 divided by 3 0 times 6 minus 3 which is equal to 9 divided by 18

which is 1 half and x1 that is x0 plus alpha 0 times r0 which is 0 0 plus 1

half times 3 0 which is 3 half this is one step okay so second iteration you

know x1 what is r1 r1 is b minus b x1 this is 3 0 minus 2 minus 1 minus 1 2

times oops sorry of course it's a vector 3 halves n0 I've forgotten the second

component 3 halves times 0 which is 3 0 minus 3 minus 3 half so the gradient

the residual is 0 and 3 half alpha 1 is 0 3 half times 0 3 half divided by 0 3

halves times 2 minus 1 minus 1 2 times 0 3 halves which is 9 quarters divided by

2 minus 3 halves minus and 3 here this is 9 quarters divided by 9 halves which

is again 1 half this means x2 is equal to x1 plus alpha 1 r1 which is 3 halves

and 0 plus 1 half times 0 3 halves so the second iteration is 3 halves and 3

quarters okay if you were to visualize this which you can if you want the true

minimalizer is at what we just say at 2 1, so 1 2 1 so the Minimalizer is here

this is X star, we have started with x0 here the first step was at 1.5 and 0

there's x1 and the second step was at well this whole half this is community, this is here

There is x2. If you were to do one more step then you would see that

this goes in this zigzag shape, it would go here and up here and try to converge to this x star here.

And that's it, that is already question number three,

and this is how you apply gradient descent to this quadratic optimization problem.