英国 Laryngograph 电子声门仪/喉头仪 EGG-D100（新款）

摘要 · 中文整理：语音情感识别仍面临诸多挑战：单模态信息的表征能力有限，离散情感标注难以刻画连续的情感过渡，多模态融合方法中模态结构差异与跨样本对齐问题依然存在。为此，本研究从数据与模型两个角度展开工作。在数据方面，构建了中文多模态数据库 STEM-E2VA，同步采集发音运动学、声学、声门信号和视频四种模态数据，覆盖七类离散情感并采用PAD连续标注；通过离散标注与连续维度标注的结合，更好地刻画同一离散情感标签下强弱情感的差异。同时，为处理PAD标注中的偏差，采用SCL-90心理问卷分析标注者的认知与情感感知，以保证数据可靠性。在模型方面，本文提出了一种融入PAD感知的多模态监督对比融合网络，采用PAD增强的混合对比损失函数优化模态内与模态间的特征对齐，并利用交叉注意力机制结合GRU–Transformer网络提取时序特征，实现多模态信息的深度融合，降低模态间差异与类间混淆。实验表明，该方法在 STEM-E2VA 上的离散情感识别准确率达到85.47%，PAD维度预测的均方根误差大幅下降，并在 IEMOCAP 上表现出优异的泛化能力，为离散与连续情感表征的融合提供了新框架。

原文摘要：There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multi-modal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators' cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU-Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.

关键词：多模态情感识别、愉悦度—唤醒度—支配度、对比学习、特征融合、多元线性回归分析

研究目标：解决语音情感识别中单模态表征能力有限、离散标注难以刻画连续情感过渡、多模态融合存在结构差异与跨样本对齐困难等问题，并提供一个同步采集声门信号的高质量中文多模态情感数据库。

主要结论：构建了同步包含 EMA、EGG（EGG-D100）、音频与视频的 STEM-E2VA 数据库；提出的 MMCL-PAD 框架在离散情感识别上达到 85.47% 的准确率，较基线提升 2.87%，唤醒度与愉悦度 RMSE 分别降至 0.145 与 0.152，声门信号使唤醒度预测误差相对纯声学模型下降 25.31%，在 IEMOCAP 上跨库准确率 74.85%。

创新点：1）四模态（运动学、声学、声门、视频）精确同步采集，并利用 EGG-D100 右通道同步音频实现对齐；2）离散标签与 PAD 连续标注相结合的分层标注机制，并用 SCL-90 分析标注者心理状态以处理标注偏差；3）基于 PAD 空间距离动态调节温度系数的混合对比损失与跨注意力 GRU-Transformer 融合网络。

后续建议：扩大发音人规模并纳入自然情感而非表演式情感，提高生态效度；进一步挖掘 EGG 波形的接触商、开商等生理参数作为可解释特征；探索模态缺失场景下的鲁棒融合策略，降低对专用设备的依赖；在更多跨语言、跨语料库上验证 PAD 引导对比学习的泛化性；公开数据库与代码以便社区复现和对比

APA 引文：Duan, S., Zhang, W., Li, L., Zhu, T., Zhao, F., Li, F., & Liang, H. (2026). PAD-guided multimodal hybrid contrastive emotion recognition upon STEM-E2VA dataset. Multimodal Technologies and Interaction, 10(4), 38. https://doi.org/10.3390/mti10040038

仪器引用：Glottal signals were collected by the left channel of an electroglottograph EGG-D100 at 44.1 kHz. ... In this study, the EGG-D100 device recorded two-channel signals: the left channel recorded the glottal waveform, whose amplitude was positively correlated with the contact area of the vocal cords; the right channel recorded the synchronous acoustic signal.

摘要 · 中文整理：得益于深度学习方法，语音情感识别（SER）已取得显著进展，文本信息的引入进一步提升了其性能。然而，很少有研究关注言语产生过程中的生理信息，而这些信息同样包含说话人特征（含情感状态）。为填补这一空白，我们开展了一系列实验，考察发声激励信息与发音运动学对语音情感识别的潜力。由于此类训练数据稀缺，我们提出了表演式情感数据集 STEM-E2VA，其中包含音频以及电子声门图（EGG）和电磁发音（EMA）等生理数据，EGG 与 EMA 分别提供发声激励与发音运动学信息。此外，我们用由语音经反演方法得到的估计生理数据替代实测的 EGG 与 EMA 进行情感识别，以探索该类生理信息在真实场景语音情感识别中的应用可行性。实验结果证实了在语音情感识别中引入言语产生生理信息的有效性，并展示了其在真实场景中的实用潜力。

原文摘要：Speech emotion recognition (SER) has advanced significantly for the sake of deep-learning methods, while textual information further enhances its performance. However, few studies have focused on the physiological information during speech production, which also encompasses speaker traits, including emotional states. To bridge this gap, we conducted a series of experiments to investigate the potential of the phonation excitation information and articulatory kinematics for SER. Due to the scarcity of training data for this purpose, we introduce a portrayed emotional dataset, STEM-E2VA, which includes audio and physiological data such as electroglottography (EGG) and electromagnetic articulography (EMA). EGG and EMA provide information of phonation excitation and articulatory kinematics, respectively. Additionally, we performed emotion recognition using estimated physiological data derived through inversion methods from speech, instead of collected EGG and EMA, to explore the feasibility of applying such physiological information in real-world SER. Experimental results confirm the effectiveness of incorporating physiological information about speech production for SER and demonstrate its potential for practical use in real-world scenarios.

关键词：语音情感识别、言语产生、激励源、发音运动学

研究目标：探究语音产生过程中的生理信息（EGG 反映的发声激励与 EMA 反映的发音运动学）对语音情感识别的贡献，并验证在无专用设备情况下用语音估计的生理信息替代实测数据的可行性。

主要结论：在 STEM-E2VA 上，融合语音、EGG 与 EMA 的三模态识别准确率达 88.42%，较纯语音 79.97% 提升 8.45%；使用 IAIF 与 HuBERT+TCN 估计的生理信息时三模态准确率为 82.69%，提升 2.72%，表明发声激励与发音运动信息具有互补价值并具备实际应用潜力。

创新点：1）构建同时包含音频、EGG 与 EMA 的表演式普通话情感数据集 STEM-E2VA；2）系统比较单/双/三模态下发声激励与发音运动学对 SER 的贡献；3）用语音反演得到的估计 EGG/EMA 替代实测数据，评估无设备场景的可行性。

后续建议：提取更具区分性的 EGG 时域/频域生理特征（如接触商、开商）；改进声门反演与发音反演模型以缩小估计数据与实测数据的性能差距；采用深度融合网络替代特征级拼接的 XGBoost 方案；在自然情感语料与跨语言数据集上验证方法的泛化能力；扩大发音人数量以降低说话人依赖性

APA 引文：Zhang, Z., Huang, M., & Xiao, Z. (2025). Speech emotion recognition with phonation excitation information and articulatory kinematics. arXiv. https://doi.org/10.48550/arXiv.2511.07955

仪器引用：EGG data were collected using the D100 electroglottography to measure changes in relative vocal fold contact area during laryngeal voice production. Two electrodes were placed on each side of the thyroid cartilage of speakers, and a high-frequency, low-amperage current was passed between them.

摘要 · 中文整理：本文设计并建立了一个包含发音运动学、声学、声门与面部微表情的普通话多模态情感语音数据库，并从语料设计、被试选择、录制细节与数据处理等方面进行了详细描述。各类信号同时标注离散情感标签（中性、高兴、愉悦、冷漠、愤怒、悲伤、悲痛）与维度情感标签（愉悦度、唤醒度、支配度）。本文通过对维度标注数据的统计分析验证了维度标注的有效性，并对标注者的SCL-90量表数据加以校验，结合PAD标注数据进行分析，以探究标注中的离群现象与标注者心理状态之间的内在关系。为验证数据库的语音质量与情感区分度，本文采用SVM、CNN和DNN三种基础模型计算七种情感的识别率。结果表明：仅使用声学数据时七种情感平均识别率约为82%；仅使用声门数据时平均识别率约为72%；仅使用运动学数据时平均识别率也达到55.7%。因此该数据库质量较高，可作为语音分析研究、尤其是多模态情感语音分析任务的重要数据来源。

原文摘要：A multi-modal emotional speech Mandarin database including articulatory kinematics, acoustics, glottal and facial micro-expressions is designed and established, which is described in detail from the aspects of corpus design, subject selection, recording details and data processing. Where signals are labeled with discrete emotion labels (neutral, happy, pleasant, indifferent, angry, sad, grief) and dimensional emotion labels (pleasure, arousal, dominance). In this paper, the validity of dimension annotation is verified by statistical analysis of dimension annotation data. The SCL-90 scale data of annotators are verified and combined with PAD annotation data for analysis, so as to explore the internal relationship between the outlier phenomenon in annotation and the psychological state of annotators. In order to verify the speech quality and emotion discrimination of the database, this paper uses 3 basic models of SVM, CNN and DNN to calculate the recognition rate of these seven emotions. The results show that the average recognition rate of seven emotions is about 82% when using acoustic data alone. When using glottal data alone, the average recognition rate is about 72%. Using kinematics data alone, the average recognition rate also reaches 55.7%. Therefore, the database is of high quality and can be used as an important source for speech analysis research, especially for the task of multimodal emotional speech analysis.

关键词：情感语音数据库、多模态情感识别、维度情感空间、三维电磁发音仪、电子声门仪

研究目标：解决普通话多模态情感语音资源匮乏的问题，构建同步包含发音运动学、声学、声门与面部微表情的情感语音数据库，并验证其标注效度与情感区分能力。

主要结论：数据库共获得 2943 条声学、声门与运动学样本及 24.5 小时面部微表情数据；SVM/CNN/DNN 基线在声学、声门、运动学单模态上的平均识别率分别约为 82%、72% 与 55.7%，证明数据库质量高，可用于多模态情感语音分析。

创新点：1）首次在普通话情感语音库中同步采集 EMA 运动学、EGG-D100 声门信号、音频与视频四种模态；2）离散与 PAD 维度双重标注，并利用 SCL-90 量表分析标注者心理状态与标注离群值的关系；3）以三种基线模型对各模态分别进行识别率验证。

后续建议：增加发音人数量与年龄分布，提升数据库代表性；引入自然诱发情感以补充表演式情感语料；对 EGG 波形提取接触商等生理参数并分析其与情感维度的关系；探索四模态联合建模与跨模态对齐方法；公开发布数据库并制定统一评测协议

APA 引文：Zhu, T., Li, L., Duan, S., Zhang, X., Xiao, Z., Jia, H., & Liang, H. (2024). Construction and evaluation of Mandarin multimodal emotional speech database. arXiv. https://doi.org/10.48550/arXiv.2401.07336

仪器引用：the glottis database recorded by EGG-D100 and the video database recorded by SONY-FDR ... Electronic glottography is a non-invasive measurement device that collects voice signals and has been widely used in the field of speech and pathological voice research.

摘要 · 中文整理：缺乏可用的情感病理数据库是研究构音障碍患者情感表达状况的关键障碍之一。本文构建了首个包含多视角信息的中文多模态情感病理语音数据库，收录29名健康对照者和39名不同程度运动性构音障碍患者，表达高兴、悲伤、愤怒和中性四种情感。全部情感语音通过自研微信小程序完成可懂度、情感类别及离散维度情感的标注。主观分析从情感辨别准确率、语音可懂度、效价—唤醒空间分布，以及SCL-90与病情严重程度的相关性等方面进行论证。在语音数据与声门数据上分别进行自动识别测试，音频数据上对照组平均准确率为78%、患者组为60%，声门数据上对照组为51%、患者组为38%，表明疾病对情感表达存在影响。

原文摘要：The lack of an available emotion pathology database is one of the key obstacles in studying the emotion expression status of patients with dysarthria. The first Chinese multimodal emotional pathological speech database containing multi-perspective information is constructed in this paper. It includes 29 controls and 39 patients with different degrees of motor dysarthria, expressing happy, sad, angry and neutral emotions. All emotional speech was labeled for intelligibility, types and discrete dimensional emotions by developed WeChat mini-program. The subjective analysis justifies from emotion discrimination accuracy, speech intelligibility, valence-arousal spatial distribution, and correlation between SCL-90 and disease severity. The automatic recognition tested on speech and glottal data, with average accuracy of 78% for controls and 60% for patients in audio, while 51% for controls and 38% for patients in glottal data, indicating an influence of the disease on emotional expression.

关键词：情感病理语音数据库、视频、功能性近红外光谱成像、运动性构音障碍、电子声门图、情感维度空间

研究目标：解决构音障碍患者情感表达研究缺乏可用情感病理语音数据库的问题，构建包含声学、声门、视频与 fNIRS 多视角信息的中文情感病理语音数据库并验证其有效性。

主要结论：数据库在情感表达与病理两方面均取得成功：标注者情感判别准确率高于 87.5%，对照组可懂度 93.8%、重度患者 67.3%；自动识别中音频情感识别对照 78%、患者 60%，声门数据 51%、38%，疾病严重程度分类最高 87%，说明疾病对情感表达产生影响。

创新点：1）首个中文多模态情感病理语音数据库，同步采集 EGG-D100 声门/声学信号、视频与 fNIRS；2）自主开发微信小程序实现可懂度、情感类别与维度的多任务标注；3）结合 SCL-90 心理量表与疾病严重程度进行相关性分析，并同时评估情感识别与疾病分类。

后续建议：扩大患者样本量并覆盖更多构音障碍病因类型；将 fNIRS 与声门信号进行多模态融合建模以提升患者情感识别率；分析 EGG 接触商、基频扰动等生理参数与疾病严重程度的关系；采用深度学习模型替代传统 SVM/随机森林以提升识别性能；开展纵向随访采集以评估康复过程中情感表达的变化

APA 引文：Zhu, T., Duan, S., Liang, H., & Zhang, W. (2023). Design, construction and evaluation of emotional multimodal pathological speech database. arXiv. https://doi.org/10.48550/arXiv.2312.08998

仪器引用：Electroglottograph (EGG-D100): to acquire synchronized acoustic and glottal signals ... subject wore the EGG-D100 sitting at around 3 m from the display screen, SONY-FDR camera was placed at about 50 cm away