Now you can see something.

So we're at the border of going from agent designs

without learning to agent designs with learning.

And yesterday, we kind of culminated

our study of reasoning with uncertainty

or deciding with uncertainty, which

is what realistic real world agents have to do,

with a very brief introduction to POMDPs.

Partially observable Markov decision procedures.

Markov decision procedures, you remember,

are these environments where we have

fully observable or deterministic sensors, which

makes the environment fully observable.

But non-deterministic actions.

You do something, but that can go wrong.

POMDPs add non-reliable sensors to that mix.

And that's essentially everything that can go wrong.

So in our example, our little 11 example,

we added an unreliable sensor, which is basically just

introducing an error into our considerations.

And there's only one thing to remember about these things,

is you can do POMDPs by doing MDP at not the level of states,

but the level of belief states.

Very simple realization that has a lot of consequences that

are both good and bad.

One of the good things is that we're essentially

getting exactly what we need.

Namely, the belief state is fully observable.

We know what we believe as an agent.

We can observe our own belief state.

And so we can actually compute on that.

We can't get below the belief state

because we know nothing in principle

about the actual state.

We can only have approximations or beliefs

over the set of possible states.

And so we can actually compute on that.

So the good thing is we can, in principle,

do MDP algorithms, one level up.

The bad news is that we only know

MDP algorithms that can deal with discrete spaces,

and the belief space isn't.

So even though we have this wonderful realization

of what we believe as an agent, we

can't do anything about it.

So even though we have this wonderful realization

that we don't have to learn anything new,

that turns out to be false because we

didn't learn enough earlier.

If we had had the theory, the full theory,