ICMC2025: 2025 International Conference on Multimodal Communication:

Data Science for Multimodal Communication 

Saturday & Sunday, 13-14 December 2025, Changsha

• Bilibili Livestream: ID: 23288623 http://live.bilibili.com/23288623
• Zoom Meeting ID: 942 7593 8736; Password: 198826
• For questions related to the conference, please write to davidyyx@qq.com.

Each presentation slot is 45 minutes, within which there is a period at the end of about 10 minutes for both Q & A and a break.

 

      

 

Local Venue: Lecture Hall Room 613, Foreign Studies College, Hunan Normal University

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science.
Committee members: Prof. JIANG Lihua, Prof. DENG Yunhua, Prof. WAN Guangrong, Dr. YANG Yuxiao, Dr. TAN Xiaojuan, Dr. ZHANG Ying.

The 2025 Hunan Normal University International Conference on Languages and Cultures has as its theme

Data Science for Multimodal Communication

Multimodal Communication—the dynamic interplay of speech, gesture, gaze, and body movement—is fundamental to human cognition, interaction, and learning. As the digital landscape generates an unprecedented volume of audiovisual data, the traditional methods of analyzing human communication are reaching their limits. ICMC2025 addresses this methodological inflection point by focusing on the convergence of cognitive science and data science.

ICMC2025 is dedicated to exploring methodologies for studying complex communication datasets. This shift from qualitative observation to quantitative, big-data analysis requires robust, open-source computational pipelines that are both theoretically grounded and practically accessible. We seek contributions that demonstrate how modern data science techniques are transforming the collection, annotation, processing, and interpretation of multimodal phenomena.

This year, the conference will highlight research focused on novel applications of data science tools in Multimodal Communication, including:

• AI-Powered Processing Pipelines: Developing and utilizing AI-based pipelines for the automated extraction and alignment of time-series data, bridging the gap between raw video content and cognitive theory.
• Quantitative Feature Extraction: Techniques for extracting precise acoustic and kinetic features, such as generating word-level alignment tiers for speech metrics (pitch, intensity) and employing pose estimation algorithms (e.g., OpenPose) to derive fine-grained kinematic features of gestures.
• Visualization and Exploratory Analysis: Methods for the spatial visualization of communication (e.g., heatmaps of gesture) and advanced exploratory data analysis that lead to new hypotheses about cognitive processes.
• Computational Reproducibility: The implementation of open-source tools and resources that ensure reliability and facilitate large-scale collaborative efforts across international consortia like the Red Hen Lab and the Multi-Data project.

There is also a Sunday Workshop:
Hands-On Introduction to Data Science for Multimodal Communication

This workshop introduces MULTIDATA, an online platform for the study of multimodal communication, and demonstrates how it can be used to analyze speech and gesture data from video. Through a focused case study, participants will be guided through each stage of the research process: formulating a research question, extracting multimodal data from a corpus, processing the data using the MULTIDATA pipeline, as well as analyzing and visualizing the results.

Conference: Saturday, 13 December 2025

Introduction

• 8:30-9:00 China Standard Time. Moderator: JIANG Lihua
• Group Photo at Tenglong Building
• Introduction by Professor JIANG Hongxin, Chair of the University Council
• Introduction to the Theme of ICMC2025 by Mark Turner

Plenary Speakers

Title: Exploring multimodal communication: from eyebrow movement detection to gesture space
Abstract: In the field of multimodal communication, understanding the nuances of human interaction requires a comprehensive examination of various layers of non-verbal cues. This talk will delve into multimodal communication, beginning with a micro-level analysis of eyebrow movements, which play a crucial role in conveying emotion and intent. Next, we will introduce rule-based gesture detection as a method for analyzing gestural communication, a technique that enables the definition and capture of a diverse range of gestures. Finally, we will expand our focus further, moving beyond fixed wrist positions to investigate the broader concept of gesture space. By exploring these dimensions, we aim to provide a richer understanding of how non-verbal communication shapes human interaction.

Title: Machine Recognition of Gesture
Abstract: How can the science of multimodal communication move much faster? In 2024, Torrent, Hoffmann, Lorenzi, & Turner published Copilots for Linguists: AI, Constructions, and Frames (Cambridge University Press). It demonstrated various ways in which computational approaches can assist the researcher, as copilots, to accelerate the progress. Copilot analytic techniques are now available that only a few months ago were unimagined. We are witnessing lightning advances in computational methods and technology for studying multimodal communication. This talk surveys potential copilots for helping a researcher analyze communicative performances involving vision, sound, manipulation of affordances, and especially gesture. We will look at possibilities for building virtual agents for gesture recognition and generation capabilities, for analyzing body kinematics, and for synthesizing multimodal communication.

Title: Field-tested Lessons on Multimodal Data Collection for AI in Kindergarten
Abstract: (Collaborators: Thorsten Piske, Andreas Maier.) Communication is inherently multimodal. In learning to communicate, young children learn language as part of a multimodal exchange. Studies on evidence-based instruction and immersion have highlighted the importance of multimodal cues (prosody, facial emotional recognition, and postures) in language acquisition. Moreover, recent advancements in machine learning have provided new tools for conducting fine-grained analysis in children’s multimodal communicative competence in kindergarten. However, empirical research on AI-supported early language learning remains limited, in part because it is challenging to collect data that meets the requirements of machine learning process in early childhood settings. Apart from technical issues, teachers are rarely provided with the training or institutional support necessary to carry out these studies. In this presentation, we share lessons learned from collecting multimodal data in a Chinese–English bilingual kindergarten as part of the kidsNARRATE project. In particular, we focus on two main aspects: first, the demands of machine learning workflows; and second, the unique complexities of early childhood educating in terms of ethical and developmental issues. These lessons are grouped into four areas: preparation, technical processing, teacher collaboration, and reflection—covering the whole cycle from consent to data integration. Lastly, we also discuss the underestimated yet essential role of teachers in ensuring data quality and parent communication, and call for greater support to strengthen their skills for AI-supported educational research.

Title: Mandarin Chinese ditransitive construction comprehension involves simulating transferring directions: evidence from saccadic tasks
Abstract: Language comprehension involves simulating event representations. However, previous research has primarily focused on discrete linguistic cues, with little attention to how syntactic constructions influence experiential simulations. This study examines Mandarin Chinese ditransitive sentences that imply either of two transferring directions of entities: “giving” (transfer from the subject to the indirect object) and “receiving” (transfer from the indirect object to the subject). We engaged native Chinese participants in saccadic tasks, in which they read ditransitive or static existential prime sentences (the preceding sentence), and then performed a target word (the following word) identification task. Experiments 1 and 2 investigated whether entity transfer is simulated in different direction. Results showed that participants responded faster when the transfer direction of the ditransitive prime sentences matched the location of the target object nouns, whereas no such difference was observed following static existential prime sentences. Experiment 3 replicated this pattern using pseudo-verb ditransitive constructions, indicating that simulation effects operate beyond specific verb semantics. These findings suggest that ditransitive constructions systematically trigger mental simulations with spatial specificity, activating implicit directional processing that facilitates identification processes for spatially congruent targets. Our study sheds light on how constructional meanings learned through usage activate embodied cognition in language comprehension.

Title: Quantitative analysis methods: synchronic and diachronic perspectives of cross-linguistic contrast
Abstract: This study mainly introduces how to apply quantitative analysis methods for corpus data processing to conduct synchronic and diachronic comparative research on cross-linguistic constructions. From a synchronic perspective, by using a combination of multiple correspondence analysis and binomial logistic regression models, we can compare the subjective and objective tendencies expressed in cross-linguistic constructions, such as the "verbalized color words+NP" construction in Chinese and English. We can also contrast the similarities and differences in the influencing factors of subjective and objective sensory tendencies in Chinese and English constructions, and explain their cognitive motivations by applying cognitive linguistic metonymy, metaphor, and construal theories. From a diachronic perspective, we calculate the collocation strength between verbs such as "pao" and "run" and their objects through "constructional collocation analysis", visualize the syntactic and semantic features of the English and Chinese "pao/run+NP" constructions through correspondence analysis, and deeply explore the diachronic development characteristics of the collocation strength between "pao" and "run" and objects, the similarities and differences in the diachronic development of the construction meanings of "pao" and "run" with objects and object types, as well as the cognitive mechanisms and complex adaptive environments of the similarities and differences. The scientific and precise processing of corpus data can ensure the reliability and validity of viewpoints and conclusions. And advanced and innovative research methods can solve complex problems that are difficult to address with traditional methods, and also expand research dimensions.

Title: Their floods and Our floods: News values of flood photo galleries of Associated Press and Xinhua News Agency
Abstract: The study investigates how floods are presented as newsworthy in the photo galleries of Associated Press (AP) and Xinhua News Agency, two news outlets with different cultural and political backgrounds. A distinction is made between international floods, that is, Their floods, and domestic floods, that is, Our floods. The data consists of around 1500 photo-caption complexes. The analysis adopts the framework of Discursive News Values Analysis. The findings show a similar tendency to highlight Negativity, Impact, Personalization and Superlativeness in presenting Their floods by AP and Xinhua, though the two differ in Proximity and Positivity. By contrast, Our floods are presented differently. Negativity, Impact and Personalization are foregrounded in AP’s presentation of floods in the United States, whereas Xinhua’s presentation of floods in China gives prominence to Negativity, Positivity, Personalization and Superlativeness. The study is significant for its attention to cross-cultural comparison and the genre of online photo galleries.

Title: Different Effects of Multimodal Representations on the Acquisition of Foreign Language Abstract Vocabulary: Evidence from Behavioral and EEG Data
Abstract: Behavioral and electroencephalographic (EEG) experiments were taken to investigate the effects of linguistic symbols and embodied perception on the acquisition of foreign language abstract vocabulary. Using previously unlearned foreign abstract emotion words as stimuli, 28 university students were divided into a linguistic symbol group and an embodied perception group to complete vocabulary encoding and recognition tasks. Behavioral results revealed that the embodied perception group exhibited higher subjective judgment and recognition accuracy than the linguistic symbol group. EEG results demonstrated that during the encoding phase, the embodied perception group showed larger late positive potential amplitudes, indicating deeper semantic processing. During the recognition phase, no significant N400 differences were observed between the two groups, but the embodied perception group exhibited more pronounced μ-rhythm suppression and enhanced θ-wave power, suggesting a distinct advantage in abstract vocabulary acquisition and more efficient retrieval.

Title: Combining AI tools and Expert Knowledge in Multimodal Processing Pipelines
Abstract: (Abstract to be added)

Title: Multimodal data science with the MULTIDATA EU platform for AI-powered video analysis
Abstract: MULTIDATA is an online platform for multimodal communication research, funded by an Erasmus+ KA220 HED project led by the University of Murcia in collaboration with Radboud Nijmegen and FAU Erlangen-Nürnberg. It provides an AI-driven pipeline for analyzing speech and gesture data from videos, supporting educational, research, and professional applications. The platform includes tutorials, guidelines, and didactic materials for interdisciplinary use of multimodal data, as well as a forum for collaboration and exchange. This presentation introduces the pipeline’s modules for automatic speech transcription, acoustic analysis, and gesture tracking through body keypoint detection, and presents ongoing research from the MULTIFLOW project, which applies these tools to create multimodal corpora and large-scale analyses of gesture–prosody interactions.

About Us

Conference Organizing Committee

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science.

Committee members: Prof. JIANG Lihua, Prof. DENG Yunhua, Prof. WAN Guangrong, Dr. YANG Yuxiao, Dr. TAN Xiaojuan, Dr. ZHANG Ying.

Hunan Normal University

Situated in Changsha, a city renowned for its historical and cultural significance, Hunan Normal University (HUNNU) is an institution of higher education designated as a national “211 Project” and “Double Top-Class Project” university. Jointly supported by the Ministry of Education and Hunan Province, HUNNU was founded in 1938 as a National Normal College (NNC), establishing it as one of the oldest normal universities in China. During the wave of university reforms in 1953, Hunan Normal College (HNC) was built upon that NNC foundation, and subsequently renamed HUNNU in 1984. In 1996, it was honored with inclusion in the “211 Project,” a prestigious initiative of the Chinese Ministry of Education to develop “100 key universities to be promoted in the 21st century.” Since 2000, HUNNU has undergone a period of significant expansion, merging with Hunan Teachers’ College, Hunan College of Politics and Law, and Hunan Medical College.

HUNNU comprises 24 colleges and offers 92 undergraduate disciplines across 11 principal categories: philosophy, economics, law, education, literature, history, science, technology, agriculture, medicine, management, and art. The university boasts six National Key Disciplines, including Ethics, English Language and Literature, Modern Chinese History, Developmental Biology, Theoretical Physics, and Basic Mathematics. Furthermore, it possesses nine Key Disciplines sponsored by the 211 Project and 22 provincial-level key disciplines designated under the 12th Five-Year Plan.

HUNNU has established partnerships with 171 universities and institutions across 41 countries and regions to foster personnel exchange and cooperation in teaching and scientific research. It has also co-established Confucius Institutes at Kazan Federal University in Russia, Wonkwang University in South Korea, and Southern Utah University in the United States.

Over its 80-year history, HUNNU has demonstrated consistent growth, even amidst the turmoil of World War II. Its faculty, across generations, have steadfastly adhered to the motto "Be humane, benevolent, excellent and diligent," working tirelessly to achieve the prosperity evident today. In recent years, driven by the “211 Project” and the “Double Top-Class Project,” HUNNU has made significant strides in discipline development, student education, faculty development, teaching research, and social service, exceeding the needs of Hunan Province in its educational, economic, and social development.

Looking forward, HUNNU embraces holistic education as its core mission, striving to become a leading comprehensive university. With distinct advantages in teacher training, it aims to achieve top-tier status in China and gain international recognition.

College of Foreign Studies, 410081 36 Lushan Rd., Yuelu District, Changsha, China

The College of Foreign Studies at Hunan Normal University traces its origins to the Department of Foreign Studies at National Normal College, founded in 1938. Its inaugural dean was Qian Zhongshu (1910-1998), a renowned scholar of Western and Chinese culture. Following Qian Zhongshu, the College benefited from the leadership of other eminent scholars, including Luo Kailan (1906-1988) and Liu Zhongde (1914-2008). Today, the College offers a first-level doctoral program in Foreign Language and Literature and hosts a research station for post-doctoral fellows. Under the leadership of Professor Jiang Hongxin, its English Language and Literature discipline has been recognized as a national key discipline. In September 2017, its Foreign Languages and Literatures discipline was admitted into the national “World First-Class Discipline Construction Project,” one of only six disciplines of its kind in China to receive this distinction.

The College comprises the Departments of English, Translation Studies, Russian, Japanese, Korean, French, and Public English. It also boasts a number of prominent research institutes, including the Hunan Center for International Cultural Communication, the Hunan Center for Sino-Russian Cultural Exchanges, the Center of American Studies, the Center of Northeast Asian Studies, the Center for Studies of British and Irish Literature, the Center of Modern Foreign Language Teaching, the Center of Cognitive Linguistic Studies, and the Center for Studies of British and American Poetry. The College publishes the Journal of Foreign Languages and Cultures, along with a corresponding Chinese journal, and supports three Confucius Institutes abroad.

The College faculty consists of 26 full professors, 44 associate professors, and numerous lecturers, of whom 51 hold doctoral degrees. The faculty includes two members of the Discipline Assessment Group under the State Council, two state-level teaching masters, and two recipients of the New Century Talent Program of the Chinese Ministry of Education. The College maintains partnerships with over 30 universities in the United States, Britain, Japan, Russia, and South Korea. Currently, it enrolls over 40 doctoral candidates, over 600 graduate students, and over 1,200 full-time undergraduates.

Adhering to the motto “international perspective, global sense, honesty, integrity, and versatility,” the College of Foreign Studies is dedicated to cultivating well-rounded and innovative talents who are both physically and mentally healthy, ethically grounded and intellectually developed, and equipped to adapt to societal changes.

Red Hen Lab™

The International Distributed Little Red Hen Lab™ is a global big data science laboratory and cooperative dedicated to research in multimodal communication. Red Hen leverages the expertise of researchers from diverse fields, ranging from artificial intelligence and statistics to linguistics and political communication, to create comprehensive datasets of parsed and intelligible multimodal communication. It also develops tools to process these data and any other data amenable to such analysis. Red Hen’s organizational structure and computational tools are designed to foster reliable and cumulative progress in the dynamic and challenging field of human multimodal communication. Understanding how humans create meaning and interpret forms necessitates this type of collaborative approach.