Welcome again to the lecture.

I want to start with a short review of what we've been doing.

We're looking into neural networks.

We have discussed the structure of a neural network.

You know that a neural network, an artificial neural network,

is a collection of neurons arranged in layers.

And above all, it's a function.

It's a nonlinear function that can map an input to an output.

The input can have a meaning like an image that you look at

or a piece of sound that you hear.

And the output can be, for example,

a label for this image or some representation of what

is the meaning of that sound.

And the most important part of this game

is you can train neural networks.

You can train them because they represent a function that

has many parameters.

What are these parameters?

The parameters are just the connection strengths

inside the neural network.

And we're talking of, say, thousands or a million

of these parameters.

And the way you train is, well, first of all,

you need lots of training examples

where, together with the input, like an image,

you have also the correct output,

the desired output for this particular piece of input.

For many, many images, if you have corresponding labels,

that would be your training database.

And then you go through all these training images

and you test how your network is doing.

And you are grading the network, so to speak.

You are evaluating how far off is the network

from the true desired output.

And so the function that tells us

what's the distance between the desired output

and the current network output, that's the cost function.

So that's a super important function.

The cost function teaches us how good is the neural network

doing.

And our aim is, of course, to reduce the cost,

to reduce the deviation.

And we will do that by updating the parameters

of the neural network.

And so how do we do that again?

Well, you could take any kind of optimization technique.

But what people are doing here is the most simple optimization

technique that you can think of.

So you're viewing this cost function

as a function of these many parameters, 1,000