How does face recognition technology work

How does face recognition work?

Most people are comfortable with facialsrecognition for use in Instagram filters and Face ID. However, this relatively new technology can feel a little creepy. Your face is like a fingerprint, and the technology behind facial recognition is complex.

As with any new technology, there are drawbacks to face recognition. These drawbacks become more and more apparent as the military, police, advertisers, and deepfake developers look for new ways to use facial recognition software.

It is now more important than ever that people understand how facial recognition works. It is also important to know what limitations face recognition has and how it will evolve in the future.

Face recognition is surprisingly easy

Before I dive into the many different mediums for face recognition, it is important to understand how the face recognition process works. Here are three uses for face recognition software and a simple explanation of how they recognize or identify faces:

  • Basic face recognition: Your mobile phone for Animoji and Instagram filters. The camera “searches” for the defining features of a face, especially a pair of eyes, a nose and a mouth. Then algorithms are used to capture a face and determine which direction it is facing, whether its mouth is open, etc. It is worth noting that this is not face detection, just software that searches for faces.
  • Face recognition and similar programs: When you set up Face ID (or similar programs) on your phone, it will take a photo of your face and measure the distance between your features. Every time you unlock your phone, it “looks” through the camera to measure and confirm your identity.
  • Identify a stranger: When an organization wants to identify a face, for security, advertising or surveillance purposes, algorithms are used to compare that face to an extensive database of faces. This process is almost identical to Apple Face ID, but on a larger scale. In theory, any database could be used (ID cards, Facebook profiles), but a database with clear, pre-identified photos is ideal.

Okay, let's go into the details. Because the "basic face recognition" used in Instagram filters is such a simple and harmless process, we'll focus entirely on face recognition and the many different technologies that can be used to identify a face.

Most face detection functions are based on 2D images

As you would expect, most facial recognition software relies entirely on 2D images. However, this is not done because 2D face imaging is extremely accurate. This is done for reasons of user friendliness. The vast majority of cameras take photos with no depth, and public photos that can be used for face recognition databases (e.g. Facebook profile pictures) are all in 2D.

Why is 2D facial imaging not particularly accurate? Well, because a flat picture of your face doesn't have any identifying features like depth. With a flat image, a computer can measure the interpupillary distance and the width of your mouth, among other things. But it cannot tell the length of your nose or the height of your forehead.

In addition, the 2D facial imaging is based on the visible light spectrum. This means that 2D facial imaging will not work in the dark and may be unreliable in funky or shady lighting conditions.

It is clear that the workaround for some of these shortcomings is to use 3D facial imagery. But how is that possible? Do you need special equipment to see a face in 3D?

IR cameras add depth to your identity

While some face recognition applications rely on it, it is not uncommon for face recognition to rely solely on 2D images and to rely on 3D images as well. In fact, your face recognition experience likely includes a dash of 3D in it.

This is achieved through a technique called lidar, similar to sonar. Essentially, face scanning devices like your iPhone blow up a harmless IR matrix on your face. This matrix (a wall of lasers) is then reflected off your face and captured by an IR camera (or ToF camera) on your phone.

Where does the 3D magic happen? Your phone's IR camera measures how long it takes for every bit of IR light to reflect off your face and return to the phone. The light reflecting off your nose naturally has a shorter path than the light reflecting off your ears, and the IR camera uses this information to create a unique depth map of your face. Coupled with basic 2D imaging, 3D imaging can greatly improve the accuracy of facial recognition software.

Lidar imaging is a strange concept that can be difficult to wrap your head around. If it helps, try imagining that your phone's (or facial recognition device) IR network is a whiteboard toy. As with a pin board toy, your face leaves a depression in the IR network in which your nose is noticeably lower than your eyes, for example.

Thermal imaging enables face recognition at night

One of the shortcomings of 2D face recognition is that it relies on the visible spectrum of light. Basic face recognition does not work in the dark for laypeople. However, this can be circumvented using a thermal imaging camera (yes, as in Tom Clancy).

"Wait a minute," you might say, "isn't thermal Imaging relies on IR light?" Yes. However, thermal cameras don't emit rays of IR light. They just see IR light coming from objects. Warm Objects emit a ton of IR light, while cold objects emit a negligible amount of IR light.Expensive thermal imaging cameras can even detect tiny differences in temperature on a surface, making the technology ideal for face detection.

There are a handful of different ways to identify a face with thermal imaging. All of these techniques are incredibly complicated, but share some fundamental similarities so we'll try to keep things simple with a list:

  • Multiple photos are required: A thermal imaging camera takes multiple images of a subject's face. Each photo focuses on a different spectrum of IR light (long, short and medium waves). Typically, the long wave spectrum provides most of the facial details.
  • Blood vessel maps are useful: These IR images can also be used to extract the formation of blood vessels on a person's face. It's scary, but blood vessel maps can be used like unique facial fingerprints. They can also be used to determine the distance between facial organs (when typical thermal imaging gives poor images) or to identify bruises and scars.
  • The topic can be identified: A composite image (or data set) is created using multiple IR images. This composite image can then be compared to a face database to identify the subject.

Of course, usually thermal facial recognition, used by the military, is not found in Khols. It's also not included with your next cell phone. Additionally, the thermal imager doesn't work well during the day (or in a generally well-lit environment), so it doesn't have many potential uses outside of the military.

Facial Recognition Limitations

We spent a lot of time talking about ... facial recognition deficiencies. As we have seen from IR and thermal imaging, some of these limitations can be overcome. However, there are still a few issues that have not yet been resolved:

  • disability: As you would expect, sunglasses and other accessories can trigger facial recognition software.
  • Posing: Face detection works best with a neutral, front-facing image. Tilting or turning the head can make face recognition difficult, even for IR-based recognition software. In addition, a smile, bloated cheeks, or any other pose can change the way a computer measures your face.
  • light: All forms of facial recognition rely on light, whether it is visible spectrum or IR light. As a result, strange lighting conditions can reduce the accuracy of face detection. That could change as scientists are currently developing sonar-based facial recognition technology.
  • Database: Face recognition will not work without a good database. With that in mind, it's impossible to identify a face that has not been properly identified in the past.
  • Data processing: Depending on the size and format of a database, it may take a while for computers to correctly identify faces. In some situations, such as police work, data processing restrictions limit the use of facial recognition for everyday applications (which is probably a good thing).

As of now, the best way to work around these restrictions is to use other forms of identification in conjunction with facial recognition. Your phone will ask for a password or fingerprint if it cannot identify your face. The Chinese government is using ID cards and tracking technology to reduce the error rate in their facial recognition network.

Scientists will certainly find a way around these problems in the future. They can use sonar technology in conjunction with LIDAR to create 3D face maps in any setting, and they can find ways to process facial data (and identify strangers) in an incredibly short amount of time. In any case, this technology has a great potential for abuse, so it is worth keeping up to date.

Sources: University of Rijeka, Electronic Frontier Foundation