What happens as listeners become speakers during face-to-face conversation?

What happens as listeners become speakers during face-to-face conversation?

Thomas Magnuson & Chris Coey

Department of Linguistics, University of Victoria (B.C., Canada)

Approaching talk-in-interaction as an integrated system of body movement and vocalization (e.g. Bavelas et al. 2000), we collected simultaneous ultrasound and external video (Fig. 1) of eight participants’ face-to-face conversations with the first author. Focusing on inter-turn intervals as well as the period immediately following the start of a turn, we sought to ascertain if there were regular patterns among body movements (including the head, trunk, and arms/hands) as well as tongue postures, swallowing, and inhalation once speech had begun. This paper presents preliminary findings for three female native English speakers.

In a given recording session, a participant watched the cartoon Canary Row (Warner Bros., 1950; see also McNeill 2005), after which recording began with the first author inviting the participant to narrate the cartoon as she remembered it while holding the ultrasound probe under her chin. Less-structured conversation followed the narrative elicitation. Video cameras (29.97p f.p.s.) recorded from two angles: one for gestures and body movement and the other for a profile shot consistent with the Palatron technique (Mielke et al. 2005). Ultrasound data were captured uncompressed at 29.97p f.p.s. from a Logiq-e portable ultrasound machine onto a Macintosh computer equipped to convert the VGA ultrasound signal to HDMI and synchronously record audio from a shotgun microphone. Postprocessing of the data was done using Vegas Pro 12 (Sony Creative 2012) for alignment and compilation (Fig. 2), then ELAN (MPI 2001–2013, Wittenburg et al. 2006) for frame-by-frame video segmentation and Praat (Boersma & Weenink 1992–2013) for audio segmentation.

Thus far, the data suggest that participants swallow differently when listening than when speaking. When listening, participants optionally brace the tongue against the alveolar ridge and anterior palate. During these intervals, unique swallowing events can occur up to four times before bracing is released. Swallowing also occurs during speech; however, swallowing does not occur more than once, and bracing intervals are shorter for all three speakers (Fig. 3). Three separate, single-factor ANOVAs with context (listening vs. speaking) as the independent factor and duration as the dependent variable revealed the duration difference to be significant for each speaker (Speaker 1: F(1, 84) = 28.82, p<.001; Speaker 2: F(1, 32) = 19.94, p<.001; Speaker 3: F(1, 63) = 6.778, p=.012). In running speech, the shorter swallowing events occur either in lieu of or immediately prior to brief inhalations, which in turn occur at fairly regular intervals at linguistically appropriate breaks (e.g. Hixon 1987).

In general, listening-related tongue bracing (and swallowing events) may co-occur with head movements (e.g. nods) and trunk movements (e.g. leaning, swiveling), as well as nasal back-channels (e.g., “hmm?”) but not, in these data, with manual gestures. Where nasal back-channels co-occur with head movements, the head tends to move before the onset of the acoustic signal and finish after it. This pattern of head >> body >> speech tends to hold for the onset of longer speech turns as well. Following the release of listening-related tongue bracing, our speakers often produced transitional utterances (e.g. “So, uhmm…”) as ‘soft-starts’ to their turns at talk. Here, too, head and body movement precedes speech. Mid-speech swallowing intervals and/or brief inhalations often occur following these transitional utterances as well. Elsewhere, once a turn at talk is fully underway, mid-speech swallowing and/or brief inhalations both take place during hold phases of manual gestures, while brief inhalations may also co-occur with stroke phases (Kendon 1980; see also Kipp 2007 for discussion of gesture phases). Manual gestures tend to begin after the onset of speech within the main portion of speakers’ turns, though not exclusively.

References

Sony Creative Software Inc. (2012). Vegas Pro. Version 12.0 (Build 563) 64-bit.. (Computer program).   htttp://www.sonycreativesoftware.com.

Bavelas, J., J. Gerwing, M. Allison, & C. Sutton (2011). Dyadic evidence for grounding with abstract deictic gestures. In: G. Stam && M. Ishino ((Eds.), Integrating gestures: The interdisciplinary nature of gesture. Gesture Studies 44. Philadelphia: John Benjamins, 49–660.

Boerrsma, P., & DD. Weenink (1992–2013). Praat: Doing Phonetics by Computer. Version 5.3.5.(Computer program). httpp://www.praat.org.

Hixon, T. J. (1987). Respiratory function in speech and song. San Diego, CA: College-Hill Press.

Kipp, M., M. Neff, & I. Albrecht (2007). An annotation scheme for conversational gestures: How to economically capture timing and form. Language Resources & Evolution 41, 325––339.

Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In: Mary Ritchie Key (Ed.), The relationship of verbal and nonverbal communication. The Hague: Mouton, 207––227. McNeill, D. (2005). Gesture and thought. Chicago, IL: University of Chicago Press.

Mielke, J., A. Baker, D. Archangeli, & S. Racy (2005). Palatron: A technique for aligning ultrasound images of the tongue and palate. Coyote Papers 14, 96–107. MPII (2001–2013).

ELAN Linguistic Annotator. Version 4.6.1. (Computer program). Nijmegen, NNL: Max Planck Institute for Psycholinguistics. htttp://tla.mpii.nl/tools/tlaa-tools/elan/.

Warner Brothers Inc. (1950). Canary row. (Animated short film). Re-released 2001 as DVD compilation by Warner Home Video: I Love Tweety. Volume 3.

Wittenburg, P., H. Brugman,, A. Russel, A. Klassmann, & H. Sloetjes (2006). ELAN: A professional framework for multimodality research. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LRREC), ELRA, Paris.