Welcome everybody to our lecture on deep learning. Today we want to go into the topic and we want to introduce some of the important concepts and theory that have been fundamental for the field.

An old saying, and I don't know who brought it up first, which says there is nothing more practical than a good theory.

Okay, today's topic will be feed-forward networks.

And feed-forward networks are essentially the main configuration of neural networks as we use them today.

So in the next couple of videos, we want to talk about first the model and some ideas behind it,

also introduce a bit of theory. One important block will be about universal function approximation,

where we will essentially show that neural networks are able to approximate any kind of function.

The stuff that works best is really simple.

This will then be followed by the introduction of the softmax function and introduction of some activations.

And in the end, we want to talk a bit about how to optimize such parameters. And in particular, we will talk about the backpropagation algorithm.

We have a search technique which is just local search, gradient descent.

So let's start with the model. And what you heard already is perceptron.

We already talked about this, which was essentially a function that would map any input, any high dimensional input using weights

and compute an inner product of a weight vector with the input.

And then we are only interested in the signed distance that is computed this way.

And you can interpret this essentially as you see here on the right hand side.

And this is a line. So this is the decision boundary here shown in red.

What you're computing with this inner product is essentially a distance, a signed distance of a new sample to this decision boundary.

And if we then only look at the sign, we will decide whether we are on the one side or the other side.

AI is not magic.

Now, if you look at classical pattern recognition and machine learning, we are still in this domain right now,

where we would typically follow a so-called pattern recognition pipeline, where we have some measurement that is converted

and then preprocessed in order to increase the quality, decrease noise.

But in the preprocessing, you essentially stay in the same domain as the input.

So if you have an image as an input, the output of the preprocessing will also be an image,

but with probably better properties towards the classification task.

Then we want to do feature extraction. You remember the example with the apples and pears.

And from these, we extract those features, which then results in some high dimensional vector space,

which is basically a vector space representation summing up the input from all sensors.

That does not show any pictures.

And in this vector space, we can then go ahead and do the classification.

Now, what we've seen in the perceptron is that we are able to model linear decision boundaries.

And this immediately then led to the observation that perceptrons cannot solve the logical exclusive or so-called XOR.

And you can see the visualization of the XOR problem here on the left hand side.

So imagine you have some kind of distribution of classes where the top left and the bottom right is blue

and the other class is bottom left and top right.

So if you look at this, this is inspired by the logical XOR function,

then you will not be able to separate those two point clouds with a single linear decision boundary.

So you either need curves or what can help you in this kind of constellation is you use multiple lines.

But with a single perceptron, you will not be able to learn to solve this problem.

Because people have been arguing, look, we can model logical functions with perceptrons.

And then if we build perceptrons and perceptrons, we can essentially build all of logic.

You can build a machine that learns to solve more and more complex problems and more and more general problems over

than you basically have solved all the problems, at least all the solvable problems.

Now, if you can't build XOR, then you're probably not able to describe the entire logic.

And therefore we will never get there.

This was a period of time where funding to artificial intelligence research was tremendously cut down

and people would not get any new grants, they would not get money to support the research.

This period became known as the AI winter.

And winter is coming.