A New Head-Probe Stabilization Device for synchronized Ultrasound and Electromagnetic Articulography recordings

A New Head-Probe Stabilization Device for synchronized Ultrasound and 

Electromagnetic Articulography recordings

F. Sigona1 , A. Stella1 , B. Gili Fivela1 , F. Montagna2 , A. Maffezzoli2 , A. Wrench and M. Grimaldi1

1Centro di Ricerca Interdisciplinare sul Linguaggio (CRIL) – University of Salento (Lecce – Italy) 

2Dipartimento di Ingegneria dell’Innovazione – University of Salento (Lecce – Italy)

3Clinical Audiology, Speech and Language Research Centre - Queen Margaret University (Edinburgh)

In the few last decades, the technology development in both imaging and position tracking techniques allowed a refined study of articulatory correlates of speech production. Ultrasound waves are largely used to investigate the tongue motion during speech (Ultrasound Tongue Imaging, UTI), due to the acoustic properties of the tongue muscles, especially at the interface between the tongue dorsum and the upper air layer. In the panorama of the position tracking techniques stands the electromagnetic articulography (EMA), which exploits the physical properties of electromagnetic induction in order to track the position of the articulators in time (cf. [1], [3], [4], [5], [7], [8]).

The above mentioned technologies have complementary characteristics: e.g., EMA has high time resolution and can track the motion of articulators outside the internal of the mouth, but it supports a limited number of recording channels, whereas UTI is non-invasive, can provide a full profile of tongue dorsum (and sometime of the tongue root too), but it has a sensibly lower sampling rate. For this reason, their simultaneous use constitutes an added value to reach a deep understanding of the articulatory correlates of speech production, e.g. to put in relation the morphology of the tongue with the articulatory landmarks on the tongue surface or other oral articulators for a deeper understanding of muscle structures coordination. In such cases, some further care is necessary. UTI typically requires a head-probe stabilization device in order to avoid misalignment during data collection that could lead to image a different tongue profile than the desired one. Various systems have been proposed and successfully used (see, among others, [2], [3], [6], [7], [10]), but none of them is adequate to support a synchronized EMA recording, due to the magnetic field distortion induced by metallic masses which would be introduced into the EMA measurement volume.

In order to overcome this problem, we designed and manufactured a new head-probe stabilization headset (Figure 1), inspired by [3] and [2], replacing the metal structure with a polycarbonate one, relying on the well-known physical properties of this material that does not interact with magnetic fields, thus not affecting at all the EMA sensors position tracking. This headset, that has a helmet form, has been tested in various speech production tasks during simultaneous speech/UTI/EMA recording sessions. In this contribution, after describing the synchronization system, we will present and discuss a series of tests performed by means of a Toshiba Aplio XV ultrasound equipment and both the AG500 and the new AG501 electromagnetic articulographs by Carstens Medizinelektronik GmbH. Provided that the headset is absolutely transparent to the EMA operation, the aim of the tests is to show that the helmet and the ultrasound probe do not significantly move over the subject, resulting in reliable ultrasound tongue images, and that these are well time-aligned to speech and EMA data.

In order to test the performance of the synchronized system, we built a series of 10 pseudo-words composed by 4 syllables and stressed on the second one (Table 1). The words differ in the primary articulators employed for the production of the consonantal segments: i.e., constrictions are produced with different parts of the tongue and/or different values of the dynamical parameters of the articulatory movements involved. Furthermore, consonants are inserted in the vocalic cycles /i-a-i/ and /o-a-o/, in order to detect different coarticulation effects due to the influences of the vocalic context on the consonantal movements.

Table 1: pseudo-words exploited to test the performances of the helmet and the synchronized UTI/EMA system.

The corpus used in this experiment is composed by target words produced in isolation and target words in YN questions (here only the former were discussed). In particular:

        − Target words produced in isolation: the subject was asked to produce 3 consequent repetitions of each word. They were                    also asked to produce them with a contrastive intonation, in order to maximize the articulatory effort employed for the production of the stressed syllable.

        − Target words in YN questions: each word was inserted in the carrier sentence “Hai visto X ?” (“Did you see X ?"), where X   is the target word. The subject was asked to produce each question one time with a surprise intonation, which should be useful to amplify the head movements during the spoken production.

The stimuli are presented in random order using the Linguometer software [11] and the entire corpus was produced 7 times by the subject. We recorded the productions of one subject, collecting 70 triplets of words in isolation and 70 questions per subject.

Figure 2: synchronized presentation and recording block diagram.

For each stimulus, the presentation equipment issues a couple of trigger pulses (begin/end stimulus) which are mixed to the speech signal and to the AG50x synchronization signal (Sybox), eventually input to the soundcard of the AG50x control server. The same speech+triggers signal is also input to an audio/video grabber together with the continuous PAL S-Video signal (25 fps) coming from the Ultrasound equipment (Figure 2). After the end of the recording session, the AG50x “calcpos” software routine can convert the EMA recorded data (250 sample/sec for AG501, 200 sample/sec for AG500) into articulator position tracking P(x,y,z,t). At this point, several Linguometer software routines can perform a first coarse time-alignment of speech/EMA/UTI data, based on the triggers timing on the audio traces, and then a second finer cross-correlation-based time-alignment and segmentation. At the end of this semiautomatic process, for each presented stimulus a speech/EMA/UTI data set is ready for whatever subsequent feature extraction (including tongue contour tracing) and analysis.EMA receiving coils were placed over the tongue at the intersection with the mid-sagittal plane (4), over upper and lower incisors (2), upper and lower lips (2), behind the ears (2), over the helmet (4), over the Ultrasound probe (1). The helmet was holding an ultrasound convex probe under the chin, in order to image a mid-sagittal section of the tongue surface.

Regarding the stability, the results showed that the new polycarbonate helmet can be adjusted to fit to the subject head and to assure the correct position of the US probe to the jaw, allowing an adequate freedom of movements of the oral articulators and of the head. In order to get a quantitative estimate of the goodness of blocking, the mutual Euclidean distance between the sensors over the probe, over the helmet and those placed behind the ears and over the upper teeth were considered. An ideal setup would require that each sensor position should not change relatively to other sensors, even if the subject produces speech in particular emphatic context, e.g. surprise intonation, which could stress the stability of the helmet to the head. In our study we found that the sensor distances were constants over time, they were varying no more than 5 mm in the worst case while the subject was naturally producing the required corpus. This implies that the US probe was not moved so that the UTI data resulted perturbed.

Figure 3: synchronized UTI/EMA signals of the pseudo-word [li.'la.li.la] as produced in isolation. From top to bottom: audio signal; tongue tip position track on Z axis; tongue body position track on Z axis; Praat textgrid, with the indication of the acustic duration of the segments; sequence of frames acquired by the UTI system. The plots are referred to the production in the time interval of the last [a-i-a] vocalic cycle of the pseudo word.

Data acquired with the synchronized UTI/EMA are particularly useful in the study of the tongue position with respect to its morphology and shape, in particular for those parts of the tongue from which the articulatory information is lost for intrinsic limits of the single technologies. Indeed, it is largely known that UTI system can't display the image of the tongue when it is extended towards the lips, because of the presence of the bones of the chin between the tongue apex and the UTI probe. For example, in producing apical consonants such [l] or [n], the articulatory information of the tongue tip gesture, and in particular the duration of the target attainment phase of the gesture, is unrecoverable from the UTI contours. This is shown in Figure 3, where the EMA tracks and the UTI frames of tonic, post-tonic, and final syllable of the pseudo-word [li.'la.li.la] are synchronized over time (cf. caption of the figure for tiers descriptions). Looking at the US frames interval 75-79, which displays the shape of the tongue in correspondence of the maximum constriction of the tongue tip for the production of the post-tonic consonant [l], the tongue apex is completely obscured by the chin and its position in the oral cavity is no more detectable. However, in the tongue tip position track as given by EMA (second tier in Figure 3), this information can still be recovered and can be directly correlated to the shape of the tongue, since both systems are synchronized in time. In this way no articulatory information about the gestures is lost. This advantage is crucial and useful for a deeper understanding of the coordination mechanisms of oral articulators and their relation to tongue muscle structures. Overall, the findings, according to our hypotheis, showed that the polycarbonate head-probe stabilization headset allows the successfully joint (time-aligned) EMA+UTI data recording without perturbations of the probe acquisition process and the EMA magnetic fields.

References

[1] Kaburagi, T., Wakamiya, K., and Honda, M. (2005), Three-dimensional electromagnetic articulography: A measurement principle, Journal of Acoustical Society of America. 118, 428–443.

[2] Miller, A.L. and Finch, K.B., (2011) Corrected High–Frame Rate Anchored Ultrasound With Software Alignment, J. Speech Lang Hear Res, 54: 471 - 486.

[3] Scobbie, J.M., Wrench, A.A., and Linden, M. van der, (2008), Head-Probe Stabilisation in Ultrasound Tongue Imaging Using a Headset to Permit Natural Head Movement, Proceeding of 8th Internation Seminar on Speech Production, Strarbourg.

[4] Stella, M., Stella, A., Sigona, F., Bernardini, P., Grimaldi, M., Gili Fivela, B., (2013), Electromagnetic Articulography with AG500 and AG501, in 14th Annual Conference of the International Speech Communication Association, Lyon, France, 25-29 August 2013: 1316-1320.

[5] Stella, M., Bernardini, P., Sigona, F., Stella, A., Grimaldi, M., Gili Fivela, B., (2012) Numerical instabilities and three-dimensional electromagnetic articulography, in Journal of Acoustical Society of America, 132 (6).

[6] Stone, M., (2005), A Guide to Analyzing Tongue Motion from Ultrasound Images, Clinical Linguistics and Phonetics, 19, (6-7), 455-502

[7] Whalen, D.H., Iskarous, K., Tiede, M.K., Ostry, D.J., Lehnert-LeHouillier, H., Vatikiotis-Bateson, E., and Hailey, D.S., (2005) The Haskins Optically Corrected Ultrasound System (HOCUS), Journal of Speech Lang Hear Res.; 48: 543 - 553.

[8] Yunusova, Y., Green, J. R., and Mefferd, A. (2009). “Accuracy assessment for AG500, electromagnetic articulograph,” J. Speech Lang. Hear. Res. 52, 547–555.

[9] Zierdt, A., Hoole, P., and Tillmann, H. G. (1999), Development of a system for three dimensional fleshpoint measurement of speech movements, in Proceedings of the XIV International Congress on Phonetic Sciences, edited by J. J. Ohala, Y. Hasegawa, M. Ohala, D. Granville, and A. C. Baile, San Francisco, CA, Vol. 1, pp. 73–76.

[10] Hueber, T., Benaroya, E.L., Chollet, G., Denby, B., Dreyfus, G., Stone, M., (2009), Development of a Silent Speech Interface Driven by Ultrasound and Optical Images of the Tongue and Lips, Speech Communication, 52(4), pp. 288-300.

[11] Grimaldi, M., Gili Fivela, B., Sigona, F., Tavella, M., Fitzpatrick, P., Craighero, L., Fadiga, L., et al. (2008). New technologies for simultaneous acquisition of speech articulatory data: ultrasound, 3D articulograph and electroglottograph, in Atti del Convegno Internazionale LangTech, pp. 81–85.

[12] Li, M., Kambhamettu, C., and Stone, M. (2005). Automatic contour tracking in ultrasound images.Clinical Linguistics and Phonetics, 19 (6-7), 545-554