How Precise is Your Forensic Analysis? (part 2)

As we discussed last week, you can compute the precision of a forensic analysis by considering the true positive and false positive rates of your forensic analysis. The precision tells you the likelihood that the image is a fake given the results of the forensic analysis. What if, after computing the precision, I tell you that the image you are analyzing was downloaded from worth1000 (a popular photo-editing competition website). Surely you are going to want to factor this into your conclusion. 

Consider a forensic analysis with a true positive rate of 80% and a false positive rate of 10%. If this forensic analysis detects evidence of tampering, then the precision is 0.8/(0.8 + 0.10) = 88.9%. A third quantity, the prior likelihood, should also be factored into this calculation. The prior is the likelihood that the image is fake or real regardless of the results of any forensic examination (e.g., an image downloaded from worth1000 has a high prior likelihood of being fake, while an image provided by the Associated Press has a relatively low prior likelihood of being fake). 

When computing the precision, the true positive and false positive rates need to be multiplied by the prior likelihood that the image is fake or real. Consider again the forensic analysis with a true positive rate of 80% and a false positive rate of 10% applied to an image downloaded from worth1000. Your prior that the image is fake is, let’s say 95%, and of course your prior that it is real is 5%. Now, the true positive multiplied by 0.95 yields a true positive of 76% and the false positive rate multiplied by 0.05 yields a false positive of 0.5%. With these new values, the precision is now 99.3%, considerably higher than the previous 88.9%.

Notice that if the prior is 50%, then it has no effect on the computation — in fact, when we ignore the prior what we are really saying is that the likelihood of an image being fake or real is equally likely. Sometimes this is appropriate, sometimes it is not.

The true and false positive rates are relatively straight-forward to measure. The prior, however, can be more tricky to determine precisely. In general, the reliability of a source (worth1000, flickr, AP/Reuters) can provide a rough measure of confidence in the integrity of the image. Another factor that will help to establish a prior likelihood is the complexity of the scene, the complexity of the purported manipulation and the number of images being analyzed. 

Even if you can’t precisely quantify the true positive, false positive, and prior likelihood, some effort should be made on placing reasonable bounds on these values in order to provide a general sense of precision.

I am often asked to analyze images. In almost all cases, I first try to determine the prior likelihood that the image was altered. This establishes an important baseline that must always be factored into an overall forensic analysis. This prior shouldn’t, of course, color your forensic examination—but it should be factored into the overall confidence of your conclusions.

Comments

[Dr. Neal Krawetz]

You wrote: "The true and false positive rates are relatively straight-forward to measure. The prior, however, can be more tricky to determine precisely."

I'm not sure if I fully agree with this. I think a lot of it can be tricky.

The problem stems from fields where a minor amount of modification is permitted. (The precise definition of "minor" is neither precise not well defined.) For example, Reuters permits minor modifications.[1] They permit a limited amount of cropping (I think it's no more than 5% off either direction), recoloring, and touch-ups. However, tools like Photoshop can add in their own modifications (unintentional by the photographer) -- like minor sharpening and recoloring. So to say that something is or is not permitted requires a context based on what is considered acceptable.

The simplest contexts are where the picture must be camera-original. However, this is hardly ever the case.

The most difficult permit a degree of acceptable modification and the determination is qualitative and not quantitative. While quantitative metrics could be created for most attributes, I don't think it is a "relatively straight-forward" metric to measure since (1) virtually every organization has a different concept of acceptable, and (2) even within an organization, there are plenty of borderline cases that may be rated as permitted/forbidden based on who does the evaluation.

Having said that, I completely agree with your general premise. Determining "real" compared to computer graphics or modified images requires a baseline description of the image and a degree of confidence for both the analysis method and the results.

[1] http://blogs.reuters.com/blog/archives/4327

[I agree that it can be difficult to quantify subjective definitions of appropriate and inappropriate manipulations. However, a forensic tool that seeks to measure a specific manipulation (cloning, resizing, lighting, sharpening, etc.) can be more easily quantified since the presence/absence of a specific manipulation is not subjective. Put another way, quantifying the precision of a forensic technique that enforces a subjective policy is difficult (and in my opinion not advisable), while quantifying an objective and well defined form of manipulation is manageable. -Hany ]