Welcome to today's lecture on activation functions and convolutional neural networks.

As you again will recognize, I'm not Professor Meier, but I'm here in his place again today.

So what did we talk about last week, just as a short recap.

So we talked about optimization strategies and how to derive loss functions in general.

And as I mentioned multiple times, this is very important for your exam.

So if you have issues understanding anything in that topic, please make sure to talk to

your tutors in the exercise and to kind of really look into these details and try to

understand also how we derive, for example, why the cross entropy kind of comes logical

from having the assumption of a categorical distribution.

Then also a little bit on the exercises, the deadline of exercise 0 and exercise 1 is due

this week.

So please make sure to submit your solution also during the exercise, but remember to

upload your solutions and your code to Stodon as well.

So we highly appreciate if not all of you submit your solution in the very last exercise

on Friday.

So please try to stick to your assigned exercise day.

Thanks.

Also from all other tutors, thanks.

Okay, so I kind of like the idea to have the five minute break last week, so we're going

to do that again approximately at half of the time just to refresh a little bit.

I'm just going to go quickly over the slides that we talked about last week.

In terms of activation functions, so we talked about the connections to biological neuron

and how a biological neuron in general works, the kind of all or nothing response scheme

of a biological neuron and how this kind of works together with the chemistry that is

going on in our brain.

So because this is important, I would like to again emphasize on this.

So also in our brain, the connection is not so much in some static construct, but it's

really in the connections between different neurons and how they interact with each other.

And also in the brain, you have excitatory responses and inhibitory responses.

So both of them kind of make up what our brain does or what the brain of any higher functioning

vertebrae or whatever works like.

So what these synapses can be looked like or what they look like is basically a feed

forward processing because they just go in one direction from the dendrites that take

information from other synapses and then transport it throughout the neurons to the next neuron

in the line.

What is important is the sum of incoming, so both inhibitory and excitatory activations,

you can think about it like spikes.

These all or nothing responses are like spikes that happen inside the brain.

So I think we can step over that.

What is important here is that the all or nothing response is like a sine function.

So there is either no response or there is a response, but there's nothing in between.

But mathematically, this is very undesirable because we don't have a gradient.

And as we've talked about last week, a gradient is pretty important, even if it's not defined

at one position, like with Drilloo, we don't have a normal gradient, for example, at position

zero.

But in general, we want to have a well behaving gradient to be sure to optimize the parameters

in our networks correctly.

Because otherwise, the back propagation scheme that we want to apply and that we want to

use to train our weights or that we want to use to calculate the gradients that we use

to update our weights is not possible.