Statistical analysis of tongue shape in ultrasound video sequences: tongue tracking and population analysis

Statistical analysis of tongue shape in ultrasound video sequences: 

tongue tracking and population analysis

Sarah Ghrenassia1 , Catherine Laporte1 , and Lucie Ménard2 

1Laboratoire de traitement de l’information en santé, École de technologie supérieure, Canada

 2Laboratoire de phonétique, Université du Québec à Montréal, Canada

Introduction and background

In this study, we developed and evaluated a set of automated and semi-automated image processing tools for the statistical analysis of tongue shape, as seen in the midsagittal view on 2D ultrasound video sequences. It is known that tongue shape and motion, though highly non-rigid, are also highly constrained: most speech patterns can be explained by a few degrees of freedom. These, in turn, are well captured by so-called active shape models (ASMs) (Cootes et al., 1995) based on Principal Component Analysis, which we use to derive the principal modes of variation in tongue shape. We fitted an ASM to a training database of 1978 tongue contours extracted from ultrasound video sequences of two adult subjects uttering [VpV] and [VtV] sequences in French (where V is one of the ten French oral vowels) shows that 95% of the shape variations are explained by three principal components. We then successfully applied the ASM framework to two different tasks related to ultrasound-based speech analysis: (1) tongue tracking and (2) phonological characterization.

Improvement of tongue tracking using ASMs

We first applied the ASM framework to the task of tongue tracking in ultrasound video sequences. Specifically, we built upon the well-known active contour approach of Li et al. (2005), which is implemented for instance within their widely used EdgeTrak software, and used our ASM to constrain the active contour to agree with plausible tongue contour shapes, as previously suggested by Hamarneh and Gustavsson (2000). As illustrated in Fig. 1, this can reduce the occurrence of tracking errors (and the time spent manually correcting them), even for utterances not included in the training database. Improvements were particularly marked in the presence of rapidly changing tongue curvature, as in the utterance [uku]. These findings corroborate previous work by Roussos et al. (2010) who also developed a shape model for ultrasound tongue tracking using PCA, but trained the model on X-ray images. In contrast, we showed that ultrasound images are sufficient to obtain a useful shape model, making the approach less invasive.

Using ASMs for phonological population analysis

We then used our ASM to describe tongue shapes in ultrasound video sequences of two adults and two 4-year-old children uttering [VpV] and [VtV] sequences in French. We fitted the ASM to tongue contours extracted from each frame in the video sequences and plotted each shape model parameter as a function of time. We coarsely aligned the curves corresponding to different repetitions of individual utterances by the same subject using the shortest utterance as a reference. Fig. 2 shows the first shape model parameter (the weight of the first principal component) as a function of time for each subject and repetition of the utterance [yty] after alignment. While the fit of the ASM was very good for adults and children alike, plotting the evolution of the parameters corresponding to each principal component showed interesting differences between adult and child populations, with adult parameters showing far more repeatability over similar utterances than child sequences, agreeing with the discussion by Smith (2010). These preliminary results demonstrate the potential of ultrasound-based ASMs as tools to reveal and analyze significant tongue shape or motion differences between populations.

Conclusions

ASMs are a potentially powerful tool both for the automated/semi-automated extraction of tongue contours from ultrasound images and for the temporal analysis of tongue shape variability among and between different phonological populations.

References

Cootes, Timothy F., et al. "Active shape models-their training and application." Computer vision and image understanding 61.1 (1995): 38-59.

Hamarneh, G. and T. Gustavsson. "Combining snakes and active shape models for segmenting the human left ventricle in echocardiographic images." Computers in Cardiology 27 (2000): 115-118.

Li, Min, Chandra Kambhamettu, and Maureen Stone. "Automatic contour tracking in ultrasound images." Clinical linguistics & phonetics 19.6-7 (2005): 545-554.

Roussos, Anastasios, Athanassios Katsamanis, and Petros Maragos. "Tongue tracking in ultrasound images with active appearance models." 16th IEEE International Conference on Image Processing, 2009.

Smith, Anne. “Development of Neural Control of Orofacial Movements for Speech.” In Handbook of Phonetic Sciences. W. Hardcastle, J. Laver, & F. Gibbon (Eds). Oxford: Blackwell, 2010