Can a person-specific face recognition algorithm be used to determine a person's race?

It's a valid question: can a person-specific face recognition algorithm be used to determine a person's race?

I trained two separate person-specific face detectors.  For each detector I used videos of the target person's face to generate positive examples and faces from [google image search for "faces"] as negative examples.  This is a fairly straightforward machine learning problem: find a decision boundary between the positive examples and the negative examples.  I used the VMX Project recognition algorithm which learns from videos with minimal human supervision.  In both cases, I used the VMX webapp for training (training each detector took about ~20 minutes from scratch).  In fact, I didn't even have to touch the command line.  Since videos were used an input, what I created are essentially full-blown sliding window detectors, meaning that they scan an entire image and can even find small faces. I then ran this detector on the large average male face image.  This average face image has been around the internet for a while now and it was created by averaging people's faces.  By running the algorithm on this one image, it analyzed all of the faces contained inside and I was able to see which country returned the highest scoring detection!


Experiment #1
For the first experiment, I used a video of my own face.  Because I was using a live video stream, I was able to move my face around so that the algorithm saw lots of different viewing conditions.  Here is a the output.  Notice the green box around "Poland."  Pretty good guess, especially since I moved from Poland to the US when I was 8.

Here is a 5 min video (VMX screencapture) of me running the "Tomasz" (that's my name in case you don't know) detector as I fly around the average male image.  You can see the scores on lots of different races.  High scoring detections are almost always on geographically relevant races.

Experiment #2
For the second target, I used a few videos of Andrew Ng to get positives.  For those of you who don't know, Andrew Ng is a machine learning researcher, entrepreneur, professor at Stanford, and MOOC visionary.  Here is the result.  Notice the green box around "Japan."  Very reasonable answer -- especially since I didn't give the algorithm an extra Asian faces for negatives.

Here is a 5 min video (VMX screencapture) of me running the "Andrew Ng" detector as I fly around the average male image.

In conclusion, person-specific face detectors from VMX can be used to help determine a person's race.  At least the two VMX face detectors I trained behaved as expected.  This is far from a full-out study, but I only had the chance to try out on two subjects and wanted to share what I found.  The underlying algorithm inside VMX is a non-parametric exemplar-based model.  During training the algorithm uses ideas from max-margin learning to create a separator between the positives and negatives.  

If you've been following up on my computer vision research projects, you should have a good idea of how these things work.  I want to mention that while I showcase VMX being used for face detection, there is nothing face-specific inside the algorithm.  The same representation is used for bottles, cars, hands, mouths, etc.  VMX is a general purpose object recognition ecosystem and we're excited to finally be releasing this technology to the world.

There are lots of cool applications of VMX detectors.  What app will you build?

To learn more about VMX and get-in on the action, simply checkout the VMX Kickstarter project and back our campaign.  

An accidental face detector

Disclaimer #1: I don't specialize in faces.  When it comes to learning, I like my objectives to be convex.  When it comes to hacking on vision systems, I like to tackle entry-level object categories.

Fun fact #1: Faces are probably the easiest objects in the world for a machine to localize/detect/recognize.

Note #1: I supplied the images, my algorithm supplied the red boxes.

Note #2: Sorry to all my friends who failed to get detected by my accidental face detector! (see below)

So I was hackplaying with some of my PhD thesis code over Thanksgiving, and I accidentally made a face detector.  oops!  I immediately ran to my screenshot capture tool and ran my code on my Mac desktop while browsing Google Images and Facebook.  It seems to work pretty well on real faces as well as sketches/paintings of faces (see below)!  I even caught two Berkeleyites (an Alyosha and a Jianbo), but you gotta find them for yourself.  The detector is definitely tuned to frontal faces, but runs pretty fast and produces few false positives.  Not too shabby for some midnight hackerdom.

Yes, I'm doing dense multiscale sliding windows here.  Yes, I'm HoGGing the hell outta these images. Yes, I'm using a single frontal-face tuned template.  And yes, I only used faces of myself to train this accidental face detector.

Note: If I've used one of your pictures without permission, and you would like a link back to your home on the interwebs, please leave a comment indicating the image and link to original.

CMU Robotics Instititue's vision finds a home at Google

Congratulations to PittPatt for their recent acquisition by Google.  PittPatt, a Pittsburgh-based startup, has its roots in CMU's Robotics Institute (where I'm currently a PhD student).  Henry Schneiderman, the CEO of PittPatt, did some truly hardcore computer vision work while doing his PhD under Takeo Kanade.

Two famous papers by Hendry Schneiderman and Takeo Kanade are the following:
H. Schneiderman, T. Kanade. "A Statistical Method for 3D Object Detection Applied to Faces and Cars". IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2000) pdf format


H. Schneiderman, T. Kanade. "Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition." IEEE Conference on Computer Vision and Pattern Recognition (CVPR 1998). pdf format


Here is what the front page of PittPatt states:

Joining Google is the next thrilling step in a journey that began with research at Carnegie Mellon University's Robotics Institute in the 1990s and continued with the launching of Pittsburgh Pattern Recognition (PittPatt) in 2004. We've worked hard to advance the research and technology in many important ways and have seen our technology come to life in some very interesting products. At Google, computer vision technology is already at the core of many existing products (such as Image Search, YouTube, Picasa, and Goggles), so it's a natural fit to join Google and bring the benefits of our research and technology to a wider audience. We will continue to tap the potential of computer vision in applications that range from simple photo organization to complex video and mobile applications.

We look forward to joining the team at Google!

The team at Pittsburgh Pattern Recognition

Perhaps Henry's success is yet another reason to come to CMU to get a vision PhD...