Computer Vision In Real Time

Face Detection and Recognition in Real-Time

Whenever we talk about Computer Vision, human face detection and face recognition are two areas which definitely needs a discussion. Although it is not an unsolved problem any more, but still having an accurate real time face detection and recognition system is very important and is applicable in many places like in Surveillance systems, Smart Home Security devices and systems, in smart traffic automation systems etc.

In this post, we will explore, various frameworks and packages, that can help us to do real-time face recognition and detection and we will compare and comment of each of these techniques and see which one works well over the other!

Approach 1 – Using the Face Recognition Python module

Out of many frameworks and algorithms that I have played around, I found the Face Recognition Python module to be the most easy to use model for doing face detection and recognition in real time. It is the simplest face recognition library that can recognize and manipulate faces from Python or from the command line.

Setting Up!

Although, using the module is very easy, but installing it is not so easy and it requires you to install the dlib package (Please note – I ran into issues when I had used the latest version of dlib – 19.20, so had to install an older version 19.19.0 to run things). But in this tutorial I am not giving the exact steps needed to setup dlib, especially if you are a Windows user (it is really a painful task!), but if anyone of you are really struggling and was not able to find a solution online, drop me a personal email and I will guide you with the exact steps to get that running! But in short, you will require a GPU system, with CUDA installed properly and CMAKE and Visual Studio build tools for Windows, then dlib and finally the latest version of the face recognition module.

Running it!

# import modules - install these modules if you don't have them
import face_recognition
from PIL import Image
import matplotlib.pyplot as plt
import dlib
dlib.DLIB_USE_CUDA
import cv2
import numpy as np

A small utility function to display images :

# Utility function to display images
def show_image(img, fig_size = None):
    print(img.shape)
    img = img.astype('uint8')
    
    if fig_size is not None:
        plt.figure(figsize=fig_size)
    
    plt.imshow(img)
    plt.show()

Set base image for doing face detection :

sample_image = face_recognition.load_image_file("data/me.jpeg")
face_locations = face_recognition.face_locations(sample_image)

print('Number of detected faces:',len(face_locations))

Now, let’s see the face region, so that if required we can use this to form bounding boxes around the detected face.

for face_location in face_locations:

    # Print the location of each face in this image
    top, right, bottom, left = face_location
    print("A face is located at pixel location Top: {}, Left: {}, Bottom: {}, Right: {}".format(top, left, bottom, right))

    # You can access the actual face itself like this:
    face_image = sample_image[top:bottom, left:right]
    show_image(face_image)
    # use the coordinates to draw bounding boxes
    # draw_bounding_box(sample_image, top, left, bottom, right)

Now, create face encodings, later we will use the face encodings generated for the face detection part!

known_encoding = face_recognition.face_encodings(sample_image)[0]
known_face_encodings = [
    known_encoding
]

known_face_names = [
    "Aditya",
]

We can create multiple, face encodings for recognizing multiple people.

Now, let’s see the real time operation.

Face Recognition and Detection in Real Time!

while True:
    # Grab a single frame of video
    ret, frame = video_capture.read()

    # Resize frame of video to 1/4 size for faster face recognition processing
    small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25)

    # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
    rgb_small_frame = small_frame[:, :, ::-1]

    # Only process every other frame of video to save time
    if process_this_frame:
        # Find all the faces and face encodings in the current frame of video
        face_locations = face_recognition.face_locations(rgb_small_frame)
        face_encodings = face_recognition.face_encodings(rgb_small_frame, face_locations)

        face_names = []
        for face_encoding in face_encodings:
            # See if the face is a match for the known face(s)
            matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
            name = "unknown"

            # # If a match was found in known_face_encodings, just use the first one.
            # if True in matches:
            #     first_match_index = matches.index(True)
            #     name = known_face_names[first_match_index]

            # Or instead, use the known face with the smallest distance to the new face
            face_distances = face_recognition.face_distance(known_face_encodings, face_encoding)
            best_match_index = np.argmin(face_distances)
            if matches[best_match_index]:
                name = known_face_names[best_match_index]

            face_names.append(name)

    process_this_frame = not process_this_frame


    # Display the results
    for (top, right, bottom, left), name in zip(face_locations, face_names):
        # Scale back up face locations since the frame we detected in was scaled to 1/4 size
        top *= 4
        right *= 4
        bottom *= 4
        left *= 4
        
        color = (0, 255, 0)
        
        if name == "unknown":
            color = (0, 0, 255)

        # Draw a box around the face
        cv2.rectangle(frame, (left, top), (right, bottom), color, 2)

        # Draw a label with a name below the face
        cv2.rectangle(frame, (left, bottom - 35), (right, bottom), color, cv2.FILLED)
        font = cv2.FONT_HERSHEY_DUPLEX
        cv2.putText(frame, name, (left + 6, bottom - 6), font, 1.0, (255, 255, 255), 1)

    # Display the resulting image
    cv2.imshow('Video', frame)
    
    try : 
        # Hit 'q' on the keyboard to quit!
        if cv2.waitKey(1) & 0xFF == ord('q'):
            # Release handle to the webcam
            video_capture.release()
            cv2.destroyAllWindows()            
            break
    except:
        break

# Release handle to the webcam
video_capture.release()
cv2.destroyAllWindows()

Now, let’s see the results in real time!

Here we go! It was able to identify me correctly and it was not able to detect Ronaldo as I did not train the system to identify Ronaldo! But it was fascinating to see how fast it was able to recognize and detect faces in real time!

To see the full code and notebook, please visit the GitHub link.

We will see other approaches to do the same, in the following sections! So stay tuned!