Visual Speech Feature Extraction

Investigators: Dr. Eun-Jung Holden, Professor Robyn Owens, Mark Barnard
Last updated: 21 November 2001.

[This is an ARC supported project.]

1. Overview

In human-machine interactions, automatic lipreading provides an alternative or additional communication channel to speech recognition.  The visual speech recognition requires extraction of features from mouth images. We propose a modified snake algorithm to extract the width and height of the mouth by combining 2D template matching technique and snakes. The inner mouth appearance is analysed by Linear Predictive Coding (LPC) technique and Higher order Local AutoCorrelation (HLAC) technique.

2. Template Matching Snakes

Amongst various techniques of controlling snakes, we chose the method of Williams and Shah, called the greedy algorithm, which has advantages in speed and simplicity over other well-known methods such as the technique of Kass's or Amini's, yet reported to produce comparable results.

The total energy is defined by the combination of continuity and curvature constraints, which correspond to first- and second-order continuity, as well as an image force, which measures the edge strength using image gradient.

The original snake algorithm however draws the snake to the contour of maximum edge strength, resulting in unstablity when teeth and tongue appear. Thus we have combined snakes with 2D template matching technique to draw the snakes to the expected outer lip contour. From the first image, the mouth locations are automatically selected and the outer lip contour template patches are extracted. Then these templates are used to draw the snakes to the similar contours by using template matching index as edge strength. The result of the template matching snake is shown below.

MPEG movie

The automatic detection of mouth location in the first image of the sequence is achieved by using Gabor wavelet responses of facial feature locations. This technique is summarised here.

The inner mouth appearance is measured by using LPC and HLAC feature extraction technique as viewed here.

Home  |  Department  | UWA