Data Science for Multimodal Communication 

There is also a Sunday Workshop:
Hands-On Introduction to Data Science for Multimodal Communication

This workshop introduces MULTIDATA, an online platform for the study of multimodal communication, and demonstrates how it can be used to analyze speech and gesture data from video. Through a focused case study, participants will be guided through each stage of the research process: formulating a research question, extracting multimodal data from a corpus, processing the data using the MULTIDATA pipeline, as well as analyzing and visualizing the results.

Conference: Saturday, 13 December 2025

Introduction

• 8:30-9:00 China Standard Time. Moderator: JIANG Lihua
• Group Photo at Tenglong Building
• Introduction by Professor JIANG Hongxin, Chair of the University Council
• Introduction to the Theme of ICMC2025 by Mark Turner

Plenary Speakers

Title: Exploring multimodal communication: from eyebrow movement detection to gesture space
Abstract: In the field of multimodal communication, understanding the nuances of human interaction requires a comprehensive examination of various layers of non-verbal cues. This talk will delve into multimodal communication, beginning with a micro-level analysis of eyebrow movements, which play a crucial role in conveying emotion and intent. Next, we will introduce rule-based gesture detection as a method for analyzing gestural communication, a technique that enables the definition and capture of a diverse range of gestures. Finally, we will expand our focus further, moving beyond fixed wrist positions to investigate the broader concept of gesture space. By exploring these dimensions, we aim to provide a richer understanding of how non-verbal communication shapes human interaction.  

Title: Machine Recognition of Gesture
Abstract: How can the science of multimodal communication move much faster? In 2024, Torrent, Hoffmann, Lorenzi, & Turner published Copilots for Linguists: AI, Constructions, and Frames (Cambridge University Press). It demonstrated various ways in which computational approaches can assist the researcher, as copilots, to accelerate the progress. Copilot analytic techniques are now available that only a few months ago were unimagined. We are witnessing lightning advances in computational methods and technology for studying multimodal communication. This talk surveys potential copilots for helping a researcher analyze communicative performances involving vision, sound, manipulation of affordances, and especially gesture. We will look at possibilities for building virtual agents for gesture recognition and generation capabilities, for analyzing body kinematics, and for synthesizing multimodal communication.

Title: Field-tested Lessons on Multimodal Data Collection for AI in Kindergarten
Abstract: (Collaborators: Thorsten Piske, Andreas Maier.) Communication is inherently multimodal. In learning to communicate, young children learn language as part of a multimodal exchange. Studies on evidence-based instruction and immersion have highlighted the importance of multimodal cues (prosody, facial emotional recognition, and postures) in language acquisition. Moreover, recent advancements in machine learning have provided new tools for conducting fine-grained analysis in children’s multimodal communicative competence in kindergarten. However, empirical research on AI-supported early language learning remains limited, in part because it is challenging to collect data that meets the requirements of machine learning process in early childhood settings. Apart from technical issues, teachers are rarely provided with the training or institutional support necessary to carry out these studies. In this presentation, we share lessons learned from collecting multimodal data in a Chinese–English bilingual kindergarten as part of the kidsNARRATE project. In particular, we focus on two main aspects: first, the demands of machine learning workflows; and second, the unique complexities of early childhood educating in terms of ethical and developmental issues. These lessons are grouped into four areas: prepa- ration, technical processing, teacher collaboration, and reflection—covering the whole cycle from consent to data integration. Lastly, we also discuss the underestimated yet essential role of teachers in ensuring data quality and parent communication, and call for greater support to strengthen their skills for AI-supported educational research.  

Title: Title: Combining AI tools and Expert Knowledge in Multimodal Processing Pipelines

Abstract:

Title: Multimodal data science with the MULTIDATA EU platform for AI-powered video analysis
Abstract: MULTIDATA is an online platform for multimodal communication research, funded by an Erasmus+ KA220 HED project led by the University of Murcia in collaboration with Radboud Nijmegen and FAU Erlangen-Nürnberg. It provides an AI-driven pipeline for analyzing speech and gesture data from videos, supporting educational, research, and professional applications. The platform includes tutorials, guidelines, and didactic materials for interdisciplinary use of multimodal data, as well as a forum for collaboration and exchange. This presentation introduces the pipeline’s modules for automatic speech transcription, acoustic analysis, and gesture tracking through body keypoint detection, and presents ongoing research from the MULTIFLOW project, which applies these tools to create multimodal corpora and large-scale analyses of gesture–prosody interactions.  

About Us

Conference Organizing Committee

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science.

Committee members: Prof. JIANG Lihua, Prof. DENG Yunhua, Prof. WAN Guangrong, Dr. YANG Yuxiao, Dr. TAN Xiaojuan, Dr. ZHANG Ying.

Hunan Normal University

Situated in Changsha, a city renowned for its historical and cultural significance, Hunan Normal University (HUNNU) is an institution of higher education designated as a national “211 Project” and “Double Top-Class Project” university. Jointly supported by the Ministry of Education and Hunan Province, HUNNU was founded in 1938 as a National Normal College (NNC), establishing it as one of the oldest normal universities in China. During the wave of university reforms in 1953, Hunan Normal College (HNC) was built upon that NNC foundation, and subsequently renamed HUNNU in 1984. In 1996, it was honored with inclusion in the “211 Project,” a prestigious initiative of the Chinese Ministry of Education to develop “100 key universities to be promoted in the 21st century.” Since 2000, HUNNU has undergone a period of significant expansion, merging with Hunan Teachers’ College, Hunan College of Politics and Law, and Hunan Medical College.

HUNNU comprises 24 colleges and offers 92 undergraduate disciplines across 11 principal categories: philosophy, economics, law, education, literature, history, science, technology, agriculture, medicine, management, and art. The university boasts six National Key Disciplines, including Ethics, English Language and Literature, Modern Chinese History, Developmental Biology, Theoretical Physics, and Basic Mathematics. Furthermore, it possesses nine Key Disciplines sponsored by the 211 Project and 22 provincial-level key disciplines designated under the 12th Five-Year Plan.

HUNNU has established partnerships with 171 universities and institutions across 41 countries and regions to foster personnel exchange and cooperation in teaching and scientific research. It has also co-established Confucius Institutes at Kazan Federal University in Russia, Wonkwang University in South Korea, and Southern Utah University in the United States.

Over its 80-year history, HUNNU has demonstrated consistent growth, even amidst the turmoil of World War II. Its faculty, across generations, have steadfastly adhered to the motto "Be humane, benevolent, excellent and diligent," working tirelessly to achieve the prosperity evident today. In recent years, driven by the “211 Project” and the “Double Top-Class Project,” HUNNU has made significant strides in discipline development, student education, faculty development, teaching research, and social service, exceeding the needs of Hunan Province in its educational, economic, and social development.

Looking forward, HUNNU embraces holistic education as its core mission, striving to become a leading comprehensive university. With distinct advantages in teacher training, it aims to achieve top-tier status in China and gain international recognition.

College of Foreign Studies, 410081 36 Lushan Rd., Yuelu District, Changsha, China

The College of Foreign Studies at Hunan Normal University traces its origins to the Department of Foreign Studies at National Normal College, founded in 1938. Its inaugural dean was Qian Zhongshu (1910-1998), a renowned scholar of Western and Chinese culture. Following Qian Zhongshu, the College benefited from the leadership of other eminent scholars, including Luo Kailan (1906-1988) and Liu Zhongde (1914-2008). Today, the College offers a first-level doctoral program in Foreign Language and Literature and hosts a research station for post-doctoral fellows. Under the leadership of Professor Jiang Hongxin, its English Language and Literature discipline has been recognized as a national key discipline. In September 2017, its Foreign Languages and Literatures discipline was admitted into the national “World First-Class Discipline Construction Project,” one of only six disciplines of its kind in China to receive this distinction.

The College comprises the Departments of English, Translation Studies, Russian, Japanese, Korean, French, and Public English. It also boasts a number of prominent research institutes, including the Hunan Center for International Cultural Communication, the Hunan Center for Sino-Russian Cultural Exchanges, the Center of American Studies, the Center of Northeast Asian Studies, the Center for Studies of British and Irish Literature, the Center of Modern Foreign Language Teaching, the Center of Cognitive Linguistic Studies, and the Center for Studies of British and American Poetry. The College publishes the Journal of Foreign Languages and Cultures, along with a corresponding Chinese journal, and supports three Confucius Institutes abroad.

The College faculty consists of 26 full professors, 44 associate professors, and numerous lecturers, of whom 51 hold doctoral degrees. The faculty includes two members of the Discipline Assessment Group under the State Council, two state-level teaching masters, and two recipients of the New Century Talent Program of the Chinese Ministry of Education. The College maintains partnerships with over 30 universities in the United States, Britain, Japan, Russia, and South Korea. Currently, it enrolls over 40 doctoral candidates, over 600 graduate students, and over 1,200 full-time undergraduates.

Adhering to the motto “international perspective, global sense, honesty, integrity, and versatility,” the College of Foreign Studies is dedicated to cultivating well-rounded and innovative talents who are both physically and mentally healthy, ethically grounded and intellectually developed, and equipped to adapt to societal changes.

Red Hen Lab™

The International Distributed Little Red Hen Lab™ is a global big data science laboratory and cooperative dedicated to research in multimodal communication. Red Hen leverages the expertise of researchers from diverse fields, ranging from artificial intelligence and statistics to linguistics and political communication, to create comprehensive datasets of parsed and intelligible multimodal communication. It also develops tools to process these data and any other data amenable to such analysis. Red Hen’s organizational structure and computational tools are designed to foster reliable and cumulative progress in the dynamic and challenging field of human multimodal communication. Understanding how humans create meaning and interpret forms necessitates this type of collaborative approach.