Image similarity comparison

I originally asked this question on cstheory.stackexchange.com but was suggested to move it to stats.stackexchange.com.

Is there an existing algorithm that returns to me a similarity metric between two bitmap images? By开发者_高级运维 "similar", I mean a human would say these two images were altered from the same photograph. For example, the algorithm should say the following 3 images are the same (original, position shifted, shrunken).

Same

I don't need to detect warped or flipped images. I also don't need to detect if it's the same object in different orientations.

Different

I would like to use this algorithm to prevent spam on my website. I noticed that the spammers are too lazy to change their spam images. It's not limited to faces. I already know there's already many great facial recognition algorithms out there. The spam image could be anything from a URL to a soccer field to a naked body.

There is a discussion of image similarity algorithms at stack overflow. Since you don't need to detect warped or flipped images, the histogram approach may be sufficient providing the image crop isn't too severe.

You can use existing deep learning architectures like VGG to generate features from images and then use a similarity metric like cosine similarity to see if two images are essentially the same.

The whole pipeline is pretty easy to set up and you do not need to understand the neural network architecture (you can just treat it like a black box). Also, these features are pretty generic and can be applied to find similarity between any kind of objects, not just faces.

Here are a couple of blogs that walk you through the process. http://blog.ethanrosenthal.com/2016/12/05/recasketch-keras/ https://erikbern.com/2015/09/24/nearest-neighbor-methods-vector-models-part-1.html

Amazon has a new API called Rekognition which allows you to compare two images for facial similarity. The api returns a similarity percentage for each face with one another and the bounding boxes for each face.

Rekognition also includes an api for both Facial Analysis (returning the gender, approximate age, and other relevant facial details) and Object Scene Detection(returning tags of objects that are within in image).

One of the great technique to calculate similarity of images is "mean structural similarity".

import cv2
from skimage import compare_ssim


img = cv2.imread('img_1.png')
img_2 = cv2.imread('img_2.png')

print(compare_ssim(img, img_2))

If you just want image similarity that's one thing, but facial similarity is quite another. Two very different individuals could appear in the same background and an analysis of image similarity show them to be the same while the same person could be shot in two different settings and the similarity analysis show them to be different.

If you need to do facial analysis you should search for algorithms specific to that. Calculating relative eye, nose and mouth size and position is often done in this kind of analysis.

Use https://github.com/Netflix/vmaf to compare the two sets of images.

First convert the images to yuv422p using ffmpeg and then run the test. Note the score difference. This can be used to tell if the image is similar or different. For this sample they both look quite similiar...

ffmpeg -i .\different-pose-1.jpg  -s 1920x1080 -pix_fmt yuv422p different-pose-1.yuv
ffmpeg -i .\different-pose-2.jpg  -s 1920x1080 -pix_fmt yuv422p different-pose-2.yuv
.\vmafossexec.exe yuv422p 1920 1080 different-pose-1.yuv different-pose-2.yuv vmaf_v0.6.1.pkl --ssim --ms-ssim --log-fmt json --log different.json
Start calculating VMAF score...
Exec FPS: 0.772885
VMAF score = 2.124272
SSIM score = 0.424488
MS-SSIM score = 0.415149

ffmpeg.exe -i .\same-pose-1.jpg  -s 1920x1080 -pix_fmt yuv422p same-pose-1.yuv
ffmpeg.exe -i .\same-pose-2.jpg  -s 1920x1080 -pix_fmt yuv422p same-pose-2.yuv
.\vmafossexec.exe yuv422p 1920 1080 same-pose-1.yuv same-pose-2.yuv vmaf_v0.6.1.pkl --ssim --ms-ssim --log-fmt json --log same.json
Start calculating VMAF score...
Exec FPS: 0.773098
VMAF score = 5.421821
SSIM score = 0.285583
MS-SSIM score = 0.400130

References How can I create a YUV422 frame from a JPEG or other image on Ubuntu

Robust Hash Functions do that. But there's still a lot of research going on in that domain. I'm not sure if there are already usable prototypes.

Hope that helps.