[Awesome@CVPR2013] Image Parsing with Regions and Per-Exemplar Detectors

I've been making an inventory of all the awesome papers at this year's CVPR 2013 conference, and one which clearly stood out was Tighe & Lazebnik's paper titled:

Finding Things: Image Parsing with Regions and Per-Exemplar Detectors.  Joseph Tighe and Svetlana Lazebnik, CVPR 2013

This paper combines ideas from segmentation-based "scene parsing" (see the below video for the output of their older ECCV2010 SuperParsing system) as well as per-exemplar detectors (see my Exemplar-SVM paper, as well as my older Recognition by Association paper).  I have worked and published in these two separate lines of research, so when I tell you that this paper is worthy of reading, you should at least take a look.  Below I outline the two ideas which are being synthesized in this paper, but for all details you should read their paper (PDF link).  See the overview figure below:

Idea #1: "Segmentation-driven" Image Parsing
The idea of using bottom-up segmentation to parse scenes is not new.  Superpixels (very small segments which are likely to contain a single object category) coupled with some machine learning can be used to produce a coherent scene parsing system; however, the boundaries of objects are not as precise as one would expect.  This shortcoming stems from the smoothing terms used in random field inference and because generic category-level classifiers have a hard time reasoning about the extent of an object.  To see how superpixel-based scene parsing works, check out the video from their older paper from ECCV2010:

ECCV2010 SuperParsing

Idea #2: Per-exemplar segmentation mask transfer
For me, the most exciting thing about this paper is the integration of the segmentation mask transfer from exemplar-based detections.  The ideas is quite simple: each detector is exemplar-specific and is thus equipped with its own (precise) segmentation mask.  When you produce detections from such exemplar-based systems, you can immediately transfer segmentations in a purely top-down manner.  This is what I have been trying to get people excited about for years!  Congratulations to Joseph Tighe for incorporating these ideas into a full-blow image interpretation system.  To see an example of mask transfer, check out the figure below.

Their system produces a per-pixel labeling of the input image, and as you can see below, the results are quite good.  Here are some more outputs of their system as compared to solely region-based as well as solely detector-based systems.  Using per-exemplar detectors clearly complements superpixel-based "segmentation-driven" approaches.

This paper will be presented as an oral in the Orals 3C session called "Context and Scenes" to be held on Thursday, June 27th at CVPR 2013 in Portland, Oregon.

scene recognition with part-based models at ICCV 2011

Today, I wanted to point everyone's attention to a super-cool paper from day 1 of this year's ICCV 2011 Conference.  Megha Pandey is the lead on this, and Lana Lazebnik (of spatial pyramid fame) is the seasoned vision community member supervising this research.  The idea is really simple (and simplicity is a plus!): train a latent deformable part-based model for scenes.  Some of the scene models look really cool, and I encourage everybody interested in scene recognition to take a look.  

A Part-based Scene Model

One of the reasons why I like this paper is because just like our SIGGRAPH ASIA 2011 paper on cross-domain image matching, they are using HOG features to represent scenes and applying these models in a sliding-window fashion.  This is much different than the traditional image-to-feature-vector mapping used in systems based on the GIST descriptor.  These types of approaches allow the detection of a scene inside another image!  Framing issues are elegantly handled by allowing the model to slide.

Scene Recognition and Weakly Supervised Object Localization with Deformable Part-Based Models. Megha Pandey and Svetlana Lazebnik. Proceedings of the IEEE International Conference on Computer Vision, 2011. Project Page [pdf]

Abstract: Weakly supervised discovery of common visual structure in highly variable, cluttered images is a key problem in recognition. We address this problem using deformable part-based models (DPM’s) with latent SVM training. These models have been introduced for fully supervised training of object detectors, but we demonstrate that they are also capable of more open-ended learning of latent structure for such tasks as scene recognition and weakly supervised object localization. For scene recognition, DPM’s can capture recurring visual elements and salient objects; in combination with standard global image features, they obtain state-of-the-art results on the MIT 67-category indoor scene dataset. For weakly supervised object localization, optimization over latent DPM parameters can discover the spatial extent of objects in cluttered training images without ground-truth bounding boxes. The resulting method outperforms a recent state-of-the-art weakly supervised object localization approach on the PASCAL-07 dataset.

Weakly Supervised Object Localization (see paper for details)