When can a photo be trusted?

Fourandsix on TwitterFollow @Fourand6
« The 3 F's of Spotting Photo Fraud | Main | Image Search for Image Forensics »
Monday
Aug132012

Lighting in Video Games

Many video games use 3-D rendering engines to generate content. These renderings are based on simplified, but physically plausible, geometric and physical models of the real-world. It can be interesting, therefore, to see examples of where a video game violates this reality and to try to understand why.

Shown below, for example, is a screen shot from the iPhone app Flick Kick Football. Take note of the shadows cast by the goal posts, soccer ball, and three players.

Shown below is a figure in which I have connected points on an object with their corresponding point on the shadow. If the shadows are consistent with a single light source, then these lines should all intersect at a single point (see “Photo forensics from shadows”). They obviously do not. Interestingly, the shadow for the goal posts (red) and players (green) are internally consistent, but are inconsistent with one another and with the ball (blue). (It may seem strange that the intersection points are below the field since the scene is certainly being illuminated from above. This is, however, consistent with a light source that is behind the camera, which seems to be the case here given the general direction in which the shadows are cast).

Shown below is an image where I have moved the ball’s shadow to a position that is consistent with the goal posts’ shadow (but not the players). In this location, the shadow seems disconnnected from the ball. This is an interesting example of where a physically correct shadow looks strange (see also “Lee Harvey Oswald”). 

There is a good chance that the game creators cheated the lighting because they felt that this led to more visually pleasing images. At first glance, the images seem perfectly credible even though the shadows are clearly inconsistent with a single light source. This is further evidence that our visual system should not be solely trusted when evaluating shadows.

PrintView Printer Friendly Version

Reader Comments (8)

See... this is why I can't enjoy video games anymore. That, and the replication in the crowd really bothers me.

August 13, 2012 | Unregistered CommenterDr. Neal Krawetz

Good point about not trusting our shadow judgement. I wonder if we are also used to, or have different expectations from, the shadows in video games because for one reason or another they are either designed to be more obvious (say in the case of the ball's shadow) or because of technological limitations aren't even realtime shadows. Especially on iOS, realtime shadows are expensive to compute and often the shadows are simply baked into the textures (for static parts of the scene) or cutout textures are projected underneath the object. This seems to be the case in this game as the shadow for the ball is bigger than the ball itself and directly underneath it. At the same time as an aid in fast gameplay it may be useful to have a simple shadow directly underneath the object for tracking purposes. It is also interesting that in this scene the dramatically elongated shadows for the statically baked goal posts indicate late afternoon whereas the ball's shadow indicates sunny high noon and the players have ambiguous partly cloudy afternoon shadows and the stands shadows seem to match the high noon shadow -- either careless design on the part of the gamemakers or perhaps, like you say, they felt it looks better this way.

August 14, 2012 | Unregistered Commentertimofei7

To start - I agree the lighting source is inconsistent for the three sources. It's probably worse than you portray. A proposed correction (based on your method in the linked page), an observation, and a question.

Correction:
I do not understand why you've drawn the red, green, and blue lines the way that you have drawn them. shouldn't the reverse ray trace be from the shadow, to the object, to the light source? (or light source, to object, to shadow). As drawn here, we are connecting object, to shadow, then to light source.

If the green or red light sources are where you show the convergence occurring, the shadows should fall to the right of the players and the goal posts, not to the left, correct? Once you connect the points per the ray trace, none of the red , or green lines converge with themselves (a red will cross a green, but that's coincidental), so the lighting model is even worse than shown.

Because I think you've drawn the ray-trace incorrectly, you have also redrawn the shadow of the ball on a false assumption. (I'm not sure where I would put it, because every element in the scene seems to have its own light source.)

The soccer ball shadow is, of course, the worst, though there might be a semi-valid game play reason for doing it that way... they wanted the angle of attack of the ball and the shadow to be the same.

Observation:
Assuming the lighting is from behind camera, and that the light source is not a point light source, but has dimension, they did get (directionally) right the hardness of the shadow. As objects get further away from the light source, the light source gets more point-like, and the shadows should get stronger, more well defined, with distance. Here they do.

Question:
Does the analysis require we assume a linear perspective projection model for the game?

My knowledge of linear algebra is going to give out at this point, but if the projection model was orthographic, or approaching orthographic (a weak perspective projection), assuming a single light source, would the shadow, to object, to light source lines still intersect? I don't know, but I suspect the ray traces might become non-linear themselves.

Again, I don't know how to prove it, but looks like the designers have chose a non-linear perspective here (or maybe just a significant telephoto perspective). Under a pinhole camera assumption, given that the goal is twice as far away, but same 'height' as player, it would imply it is twice as tall as the players if you put the players at the same distance.


[With respect to the basic lighting constraint, the lines are drawn to connect a point on a shadow to its corresponding point on the object (or vice versa). Multiple such lines are then extended, and their intersection specifies the projection of the lights source. The constraints shown above are drawn according to this simple rule.

With respect to the shadows themselves, assuming a reasonably distant light source, the softness of the shadow will depend primarily on the distance between the object and the surface onto which it projects (see, for example, "A Shadow's Penumbra").

The analysis does assume a model of linear perspective. Under a model of orthographic projection, the shadow constraints for a single light source would all be parallel (you could still detect a violation because shadows from multiple light sources would yield non-parallel constraints). The most common non-linearity introduced into an imaging model is lens distortion, which you can see is not present here because straight lines (goal posts) are being imaged as straight lines. The scene, as far as I can tell, has been imaged under a model of linear perspective (albeit with physically implausible lighting). --Hany]

August 20, 2012 | Unregistered CommenterJason

Hany - Thank you for your reply.

On the lighting constraint methodology:
"the lines are drawn to connect a point on a shadow to its corresponding point on the object (or vice versa).Multiple such lines are then extended, and their intersection specifies the projection of the lights source".

I do not dispute connecting object to shadow. My question is 'in which direction do you extend the ray to find the light source?" I maintain the method is only valid if you extend from shadow, through object, to light source (as you have done in the PFS article, Photo Forensics of Shadows), which I will call the SOL method. The method is invalid if you extend from object, through the shadow, to the projected light source, OSL method, as you've done here in the soccer example.

I use your PFS article illustration as my feeble proof to show your method cannot be indifferent to the direction in which you extend the lines, that is, OSL is not equal to SOL. (Ignore the fake bottle in this analysis).
1. You show shadow->object->light line (SOL) as converging to a single point (call it L). So assume a set of SO pairs for which the SOL method converges to a point L.
2. For this same set of SO pairs, if I then extend the lines from object->shadow-> and onwards to light (OSL) (ie. in the other direction), these lines cannot intersect at a different point L', since, by construct, they are radially eminating from the point L.
3. Therefore, for any case in which the SOL method converges, the OSL method must give a non-convergence answer, and vice-versa. Therefore the method cannot be invariant to the direction in which you choose to extend the ines.

(Yes, there is a case where the SOL lines are all parallel, meaning the OSL lines are parallel and therefore neither converges to a point, in this case both correctly give the 'not a single point light source' output).

I argue that the SOL version (as shown in PFS article) is correct, and that OSL version (in the Soccer example) is incorrect. In the OSL method, you extend from object, to shadow, to presumed light source, thus projecting the shadow between the light source and the object. That does not make a whole lot of sense to me. The SOL method preserves the relationship that the shadow is a projection of the object away from the light source.

So what happened in the soccer example:
As discussed above, unless parallel, then any two of the SO lines will have to converge under either OSL or SOL, In the soccer example the OSL lines were the ones that happened to converge and you chose to illustrate those. In the case of the three blue lines, the fact that they all converge at the same spot under OSL is either coincidence or a derivative effect of the faulty lighting model for the game. They need to converge under SOL to be a valid point light source (under a point light source assumption).

Just because the lines happened to converge, does not mean that the methodology was correctly applied. They need to be extended per the SOL method, in the other direction (up and to the right), where they won't converge to a common light source.


On the shadow:
I was specifically refering to the distance between light source and object. I was not making your assumption "assuming a reasonably distant light source". Specifically I was thinking of a moderate distance, but very large light source. It is reasonable to expect that a stadium does not have a single point light source, but is illuminated by a bank of lights that is the width of the field or wider. They've over-emphasized the effect (because in a real stadium you have very large bands of lights on both sides to diminish the shadows, and the object to light distance between the players and the goal here isn't that big a difference if we assume the lights are at the opposite goal, not midfield), but if the soccer players were significantly closer to the bank of lights than the goal, they'd have a more diffuse shadow, because the incident light is coming from a larger range of angles. (see wikipedia for "Soft Light" or here for a drawing of the effect I was referring to.)

[You are right that you must consider extending the lines in both directions. The forward direction (shadow to object) corresponds to the case when the light is in front of the camera. The backward direction (object to shadow) corresponds to the case when the light is behind the camera (as in the soccer example). (Note that if the light is in the same plane as the camera, then the lines will be parallel). If either the forward or backward constraint yields a consistent intersection, then the lighting is valid -- and except for the parallel case, at most one of these constraints can be valid. In the example above, I've only drawn the backward constraints.

Regarding the shadow, if the scene were illuminated with multiple lights then we would expect to see multiple shadows for each object (unless the lights are narrowly tuned spotlights). Having said that, this shadow analysis assumes a single local or distant light source that is close to a point light source. Care must be taken to ensure that this is the case before applying any analysis. --Hany]

August 21, 2012 | Unregistered CommenterJason

I yield! I agree it depends on the light to camera orientation. I can't yet wrap my head around exactly how the ray trace works with respect to the perspective transformation, but I've played with a few real pictures and agree it works as you describe. I suspect it is probably the same phenomena as thinking about the focal point of a lens. A light behind the film plane spreads out until it gets to the camera plane, then, due to camera perspective, is bent back to a converging path. (no need to reply, just thinking out loud).

On the shadow -> I'm coming from a photographer's perspective. A bank of a 100 stadium lights can be functionally equivalent to one a big light source -> depending on the distrance, all those 100 individual shadows will average together, getting you a feathered shadow closer in (like on the soccer player), but a harder shadow further away. Practical application is the LED light panels of 100s of lights that are starting to replacing the classic soft-box modifier for a single light in video applications.

August 21, 2012 | Unregistered CommenterJason

the question about source direction jason proposed confused me two hours too, and now I get it by make a experiment under sunshine. Thank you. And I have the question about the relationship between the position of the camera, the shadow, the player, the source and the interaction points. Is there a function between them?

[For simplicity, let's assume a scene illuminated by a distant light source (the sun). In the 3-D world, the direction and shape of shadows are determined by the position of the light source, the position and shape of the objects casting a shadow, and the shape of the surface onto which the shadows are cast. Again in the 3-D world, virtual lines that connect points on the shadows to their corresponding points on the objects will be parallel (because the light source is very far). When this 3-D scene is imaged, these virtual lines will converge to a single point (similar to a vanishing point generated from parallel lines, e.g., railroad tracks). This point is the projection of the light source into the image plane, and therefore does depend on the location of the camera. So, yes, there is a relationship between all of these things that can be described using the basic rules of perspective projection. --Hany]

August 25, 2012 | Unregistered CommenterFrank zeng

Thank you for patience reply. I download your book digitalimageforensics and study now. It's full state of art idea and really helpful. And I find there may be a minor mistake in the page35 " Bayes rule:P(B|Ak) = ..." . The suffix of the Sum in the denominator should not be L. See http://en.wikipedia.org/wiki/Bayes'_theorem

[The conditional probability P(B|Ak) is correct -- if you email me (farid@fourandsix.com) we can discuss off-line. --Hany]

August 27, 2012 | Unregistered CommenterFrank zeng

Seems to be an interesting game. I must play this. We I also like FIFA games.

September 30, 2014 | Unregistered Commenteracheter far cry 4

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>