A recent paper from Google describes a new image search algorithm that ranks images based on their visual similarity. This NYTimes article and this give a good introduction to the algorithm. I will try to give an overview of their approach:

  • They extract local descriptors (SIFT descriptors) on the images.
  • Measure of similarity between two images is defined as the number of interest points (descriptor vectors) shared between the two images divided by their average number of interest points.
  • The similarity between images is considered as probabilistic visual hyperlinks (this is necessary as there are no actual links between the images) and this leads to using the PageRank algorithm for ranking.

The above ranking method can be interpreted as finding multiple visual themes and their strengths in a large set of images and using this for ranking them. An example from the paper is shown below. There are many comic representations of the painting MonaLisa and all of them are based on the original painting. The original painting will contain more matched local features than others (and hence will be rated as having a stronger visual hyperlink). As seen in the image below, the center of the graph contains images corresponding to the original version of the painting.

The authors of the paper above have posted some clarifications about the paper here.

On a similar not, came across a good talk on Image retrieval, especially semantic image retrieval.

Using Statistics to Search and Annotate Pictures -> Gives a brief introduction to image retrieval (query by sketch, query by example) followed by the concept of semantic image retrieval.