Music

Alright, welcome to Multimedia Security last week.

So today I would like to talk about a different class of image forensics methods, so called physics based methods.

Physics based image forensics.

This is a topic that I like a lot, but it does not receive a lot of attention in the forensics community.

So the overall motivation is, so why physics based?

The goal of physics based methods is to validate specific physical properties in the scene.

For example, we have direct illumination like what we have here, and the person is standing on the ground,

then there should be a shadow somewhere at the feet of that person.

And the direction of the shadow, it should be like on the other side of the sun, loosely speaking.

For example, under direct illumination from a single source, let's be a little bit more precise to avoid complications,

if a person has a shadow that is on the other side, then the light source.

So far so simple.

But if we think about this, we already have a bunch of physical assumptions in there.

Incident light source.

This allows us to examine the direction of incident light, or the spectral distribution, so the color, spectral distribution.

Of light.

We can make an assumption that we are located on a ground plane, for example, so we have a clearly recognizable ground plane,

and we can look at the shadows cast onto that plane, so cast shadow direction on plane,

and cast shadow height, or length, I don't know, shadows have a length, right, so it's a cast shadow length on that plane that we can measure.

The different things are optical effects, so talking about measuring lengths, we can also use,

make clearly recognizable vanishing points that should be consistent.

What are vanishing points?

Yeah, okay, okay, this is not the computer vision lecture, it's fine.

Yes, it's a concept that comes essentially from perspective projection, we always have perspective projection in our normal type cameras,

and when we take a picture of a building and we're standing in front of that building, or at some corner of that building and are taking a picture of that building, it might look like this, yeah.

Sorry, I'm messing up.

So imagine there are windows here, yeah,

when you're naively looking at that picture, it looks as if the upper lines of these windows, for example, you have right angles everywhere,

but if you would connect these with lines, it looks as if these lines would converge in some point, not necessarily within the image, but somewhere outside of the image, yeah.

So these are the vanishing lines and the point where they connect are vanishing points, and the same thing also, so that, I mean we are in 3D, so there are also three directions of vanishing lines.

So, and now you could say if I have a picture with a couple of buildings, they all have right angles everywhere, yeah, then you could compute on each of these buildings the vanishing lines,

where these vanishing points are, and then use this as a consistency measure, assuming that if the picture is a composite from multiple sources, that the vanishing lines don't fit together, yeah,

that would be like a typical, like optics-motivated, physics-based criterion, yeah.

But you need this, like the clearly recognizable, sorry, vanishing lines probably, more accurate than vanishing points, so the clearly recognizable vanishing lines, so this means it only works in man-made environments,

it doesn't help a lot in nature because there you don't have these right angles which are so necessary in order to observe the perspective distortion here.

Okay, so the same thing here, these assumptions, they don't work in every situation, yeah, a single dominant light source, I think this is an assumption that is okay if you are outdoors, you have the sun,

so if you are outdoors and you have a clear sky, then it's fine, yeah, it doesn't work, like in the winter weather here, like with a cloud cover, for example, it's, you would not call this a single light source,

it's still just one sun, but there's so much interreflections that it's like one large super extended light source, yeah, and then that's typically a violation against this assumption.

Also indoor scenes are complicated because typically you have more than one light source indoors, like for example you have light coming through the window and you have a local light here from the ceiling and ta-da,

so you have two lights, or you have some interreflections that's also a source of confusion.

So these physics-based methods, the reason why they are not so popular is that they always come with assumptions of this type, yeah, so assumptions on the scene content, on the content of the scene, and they are only applicable if there's a specific situation in the scene

that can be exploited, but then, but if that is the case, then it's cool, because then you have, like, but then you have a specific property that you can actually measure from that image

and you sort of know the ground truth from your physical assumptions on the scene. So, strong assumptions

are our main limiting factor for application.

So the, yes, please?

What would be an example for a not really recognizable ground frame?

Oh, yeah, it's maybe not so, not so exact this expression, but you could have a situation like where you have a picture of a person, for example,

not on the stones here, but instead on this, like, earth-like ground next to it, yeah, and then you're trying to measure the length of the shadow and you wonder, is it consistent with other persons,

but you're not really certain that you have a perfect plane there, or it could be, it could have some local curvature, and then your measurement is not really an indication,