Navigation
Overview
Center for Cognitive Science Home

 

ICMC2024: 2024 International Conference on Multimodal Communication:

Disruption in Education: Multimodal Communication in Education, Research, and Learning 

Saturday & Sunday, 14-15 December 2024, Changsha

View the recordings of the plenary talks. Conference handbook. PDF download

Each presentation slot is 45 minutes, within which there is a period at the end of about 10 minutes for both Q & A and a break.

 

      

 

Local Venue: Lecture Hall Room 613, Foreign Studies College, Hunan Normal University

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science.
Committee members: Prof. JIANG Lihua, Prof. DENG Yunhua, Prof. WAN Guangrong, Dr. YANG Yuxiao, Dr. TAN Xiaojuan, Dr. ZHANG Ying.

The 2024 Hunan Normal University International Conference on Languages and Cultures has as its theme

Disruption in Education: Multimodal Communication in Education, Research, and Learning

Multimodal Communication is at the heart of education, research, and learning in colleges and universities. Most assessment (essays, presentations, etc.) are done via communication, and we teach multimodal communication (e.g., second language acquisition; such as the teaching of English in China). But all of our traditional practices are being disrupted now by new methods and new platforms.

There is also a workshop: Disruption in Practice

Time: Sunday morning, 15 December 2024, 9-11am China Standard Time
Location: Room 611, Zhonghe Building, which is next to the College of Foreign Studies.
Workshop ZOOM ID: 992 9292 8048; Password: 134804

Plenary speakers will hold an open and wide-ranging discussion about actual changes taking place in
Education, Learning, and Research in specific universities and cities.
Members of the audience are invited to join the conversation and give additional reports.

Conference: Saturday, 14 December 2024.

Plenary Speakers

  • 8:30-9:00 China Standard Time. Moderator: JIANG Lihua (Sat, 14 Dec 2024 at 00:30 UTC; Fri, 13 Dec 2024 at 19:30 EST; Sat, 14 Dec 2024 at 01:30 CET; Sat, 14 Dec 2024 at 00:30 GMT; Fri, 13 Dec 2024 at 16:30 PST.)
  • 9:00–9:45 China Standard Time. Moderator: WAN Guangrong    
    Michael Schoop. Senior Vice-President, Talent, Greater Cleveland Partnership. Previously Vice-President, Cuyahoga Community College, President of the Metro Campus of Cuyahoga Community College.
    Title: Disruption and Realignment in U.S. Higher Education in the New World of Work
  • 9:45-10:30 China Standard Time. (Fri, Dec 13, 2024 at 8:45 pm EST; Fri, Dec 13, 2024 at 10:45 pm BRT; Sat, Dec 14, 2024 at 1:45 am UTC.) Moderator: WAN Guangrong
    Tiago Torrent. Professor of the Graduate Program in Linguistics and the head of the FrameNet Brasil Lab at Federal University of Juiz de Fora, Brazil. Research Productivity Grantee of the Brazilian National Research Council for Scientific and Technological Development. Professor of the Graduate Program in Linguistics, Federal University of Juiz de Fora, Brazil.
    Title: Reframing Education in the Age of Copilots: multimodal language capacities for an ever-changing AI-oriented world
  • 10:30-10:40 China Standard Time. Coffee & tea break.
  • 10:40-11:25 China Standard Time. Moderator: TANG Yanling
    CHEN Fei Professor, School of Foreign Languages, Hunan University.
    Title: The Influence of Audiovisual Cross-Modal Integration on Speech Perception and its Implications for Speech Training
  • 11:25-12:10 China Standard Time. Moderator: TANG Yanling
    Renata Geld. Founding Director, Center for Cognitive Science, Associate Professor, Co-head MA in AppCogSci, cognitive science, cognitive linguistics, language education, interdisciplinarity in HE, creativity and creative ecosystems.
    Title: Learning as an inherently multimodal process: cognitive science of education
  • Lunch Break 12:10-13:45 China Standard Time
  • 13:45-14:00 China Standard Time. (Sat, 14 Dec 2024 at 05:45 UTC; Sat, 14 Dec 2024 at 00:45 EST; Sat, 14 Dec 2024 at 06:45 CET; Sat, 14 Dec 2024 at 05:45 GMT; Fri, 13 Dec 2024 at 21:45 PST.) Moderator: CHEN Minzhe
    Mark Turner. Institute Professor and Professor of Cognitive Science, Case Western Reserve University.  Co-director, the International Distributed Little Red Hen Lab.
    Title: Collaborating with Synthetic Agents
  • 14:00-14:30 China Standard Time. Moderator: CHEN Minzhe
    Martin Woesler "Xiaoxiang Scholar" Distinguished Professor and Director of the Jean Monnet Research Centre of Excellence on the Digitalization of University Teaching, Foreign Studies College, Hunan Normal University. Research fields include use of ai in foreign language acquisition, cognitive load theory, appropriateness theory, comparison of ai/human teachers, field work to identify the role of different modals/senses, e.g. by field work with learners with disabilities.
    Title: Multimodal Learning from the perspective of the cognitive load theory in the age of AI
  • 14:30-15:15 China Standard Time. Moderator: CHEN Minzhe
    DENG Yunhua Professor, Foreign Studies College of Hunan Normal University. Head of the linguistics team in the English department.
    Title: Application of Large Language Models in Language Research
  • 15:15-15:25 China Standard Time. Coffee & tea break.
  • 15:25-16:10 China Standard Time. Moderator: TAN Xiaojuan
    Mateusz-Milan Stanojević. Full Professor at University of Zagreb, Faculty of Humanities and Social Sciences
    Title: Multimodal Communication in Learning, Teaching, and Research through the Cognitive Linguistic Lens: Understanding the Disruption
  • 16:10-16:55 China Standard Time. Moderator: ZHANG Ying
    YANG Yuxiao Associate Professor and Director of the Linguistics Faculty, Foreign Studies College, Hunan Normal University.
    Title: The Application of Multi-modal Approaches in L2 Speech Learning in the Age of AI
  • 16:55-17:10 China Standard Time. Moderator: WANG Yuqing
    Concluding remarks.    Mark Turner. Case Western Reserve University.

Detailed Descriptions of Plenary Talks

Michael Schoop is Senior Vice-President, Talent, Greater Cleveland Partnership. Previously Vice-President, Cuyahoga Community College, President of the Metro Campus of Cuyahoga Community College.

Title: Disruption and Realignment in U.S. Higher Education in the New World of Work

Abstract: Demographics and economics are remaking U.S. higher education. Declining enrollment, rising costs, and questions about the relevance of traditional degrees are reshaping the landscape. At the same time, the U.S. economy is evolving rapidly, with businesses facing a growing skills gap and labor shortages, particularly in critical sectors like technology, healthcare, and manufacturing. The key question: how can higher education and business work together to bridge this gap?
Higher order skills are essential to the growth of modern economies—especially in the age of synthetic intelligence. As the world of work is transformed, the workforce must constantly reskill and upskill. How can universities provide flexible, industry-aligned programs to support this ongoing learning? What role should businesses play in fostering continuous education for their employees?
This talk explores the disruption in a market for U.S. higher education divided between prestige and economic mobility. It highlights the "last mile" problem-the gap between credentials and learning that has economic value-caused by fragmented systems and misaligned stakeholders. The challenge: how might we use data and artificial intelligence to we ensure that higher education is better connected with economic demands and meets the needs of the U.S. workforce?

     

Tiago Torrent is Professor of the Graduate Program in Linguistics and the head of the FrameNet Brasil Lab at Federal University of Juiz de Fora, Brazil. He is also a Research Productivity Grantee of the Brazilian National Research Council for Scientific and Technological Development.

Title: Reframing Education in the Age of Copilots: multimodal language capacities for an ever-changing AI-oriented world

Abstract: Schools are intended to guide pupils in their transition from family life to social and work life. Throughout the years, schools and universities have – not without some delay – resonated the changing needs of society, conforming their methodologies and approaches to education to match the expected abilities their alumni should have to be successful in the world they would encounter after graduating. The current pace of technological innovation has been challenging one foundational assumption of teachers and professors: that the world their students will find when leaving schools and universities will be the same they experienced when they started their education. In this talk I revisit language capacities that have been populating school and university curricula for the past century and discuss the extent to which the Age of Copilots imposes changes to such capacities. In particular, I address the growing importance of collaborative creative and editorial practices, highlighting the potential of synthetic agents to augment students' proficiency in multimodal communication practices.

 

CHEN Fei is a Professor and Ph.D. supervisor at the School of Foreign Languages, Hunan University. His primary research areas include experimental phonetics, pathological linguistics, and neurolinguistics.

Title: The Influence of Audiovisual Cross-Modal Integration on Speech Perception and its Implications for Speech Training

Abstract: This report examines the influence of audiovisual cross-modal integration on speech perception and its implications for speech training. The first section explores the effects of audiovisual integration on speech processing, focusing on stimulus factors (such as the impact of visual cues like facial expressions and gestures, as well as speech rate on Mandarin tone perception) and listener factors (including cross-modal speech processing mechanisms in individuals with hearing impairments using cochlear implants or hearing aids). The second section investigates the role of audiovisual integration in speech training, covering areas such as second-language pronunciation training and speech rehabilitation. Through these studies, we delve into the potential of audiovisual integration in enhancing speech perception and training, offering new perspectives for the development of speech learning and instructional methodologies.  

 

Renata Geld is Founding Director, Center for Cognitive Science, Associate Professor, Co-head MA in AppCogSci, cognitive science, cognitive linguistics, language education, interdisciplinarity in HE, creativity and creative ecosystems.

Title: Learning as an inherently multimodal process: cognitive science of education
Abstract :Students, irrespective of age, learn best when they can engage with information through multiple formats, enhancing understanding, retention, and the ability to apply knowledge. Multimodal learning is far richer than merely presenting information in different sensory formats; it taps into the interconnected and dynamic processes of the mind, where ideas are synthesized, visualized, felt, and embodied. Learning is an active, imaginative, and deeply personalized process. In other words, learning is inherently multimodal. Real learning involves the construction of mental connections, the visualization of concepts, and personal engagement with material that is both emotional and embodied. These cognitive processes enable learners to relate to new ideas, imagine applications, and draw on prior knowledge to create networks of meaning. While learning through multiple modes is natural in the real world, classroom environments often fall short in providing these rich, varied experiences. To foster deeper, more meaningful engagement, educational approaches must shift toward more authentic, multimodal learning experiences.
In modern education, authenticity and experience must become imperatives. Learning experiences should encompass hands-on, real-world tasks as well as technologically driven contexts, contributing to a dynamic form of experiential learning. Authentic learning environments, whether grounded in real-world engagement or enhanced by immersive, technology-driven simulations, promote active participation, reflection, and the meaningful application of knowledge. Furthermore, to fully prepare students for the complexities of the 21st century, it is increasingly critical to integrate the humanities and social sciences with STEM, cultivating a multidisciplinary approach that reflects real-world challenges. By integrating these diverse contexts and perspectives, we equip students with the adaptability, insight, and holistic understanding necessary to navigate a rapidly changing world.
In this talk, I will illustrate this approach with examples from a new graduate program in Applied Cognitive Science that was designed and accredited at the University of Zagreb. This program addresses the urgent need to integrate the humanities and social sciences with technical fields and STEM, providing students with a well-rounded, interdisciplinary foundation. Structured around the principles of multimodal and experiential learning, the program immerses students in real-world challenges and technology-enhanced environments to foster deep cognitive engagement. I will discuss how the program was developed, emphasizing key aspects of its curriculum, which combines hands-on projects, interdisciplinary collaboration, and immersive simulations. Together, these elements equip students with the skills and adaptability needed to thrive in today’s complex educational and professional landscapes.  

 

Mark Turner is Institute Professor and Professor of Cognitive Science, Case Western Reserve University;  Co-director, the International Distributed Little Red Hen Lab

Title: Collaborating with Synthetic Agents

Abstract: Throughout Education, Research, and Learning, we now collaborate and communicate with synthetic agents—AI, robots, computational systems. These synthetic agents can use full multimodal communication. Communication depends on common ground. How can we establish common ground with synthetic agents? How can synthetic agents be used as part of teams, in distributed fashion, across networks? How can we trust synthetic agents? What is the future of multimodal communication in a university filled with synthetic agents?

 

Martin Woesler is "Xiaoxiang Scholar" Distinguished Professor and Director of the Jean Monnet Research Centre of Excellence on the Digitalization of University Teaching, Foreign Studies College, Hunan Normal University. Research fields include use of ai in foreign language acquisition, cognitive load theory, appropriateness theory, comparison of ai/human teachers, field work to identify the role of different modals/senses, e.g. by field work with learners with disabilities

Title: Multimodal Learning from the perspective of the cognitive load theory in the age of AI

Abstract: This paper introduces some theoretical background of ongoing field studies on ai and human teaching at European and Chinese universities, including virtual learning environments. According to dual-coding theory, humans process visual and auditory information separately (at the initial stage) in visual channel (eyes) and auditory channel (ears) and integrate the resources subsequently. If working memory resources can be allocated to these two channels, the working memory load can be spread or shared, resulting in lower cognitive load (Sweller 1988) experienced by learners. In general, a more holistic learning triggering several senses like video, audio, touch, smell, and senses for space, time etc. and working memory resources can be allocated to several channels, the working memory can be spread, resulting in lower cognitive load experienced by learners. Not only the cognitive load is lower, also the remembering and associating of the different sensual/modal remembrances into the big picture of the stuff learned ("integration") goes far easier if several different impressions contribute. Therefore, a reduction of channels like in on-screen teaching during the pandemic represents a higher cognitive work load and this can be equalized by enhancing virtual learning environments to match real-life student-teacher situations with their diverse stimuli for different channels. Accordingly, learning with an ai chatbot becomes more effective when the chatbot is humanized (antropomorphized) and when it stimulates diverse senses (modals). This is one of the hypotheses we try to proof with our field research carried out at different universities in Europe and China. However, we are also interested in identifying other factors playing into the success or failure of ai teaching vs. human teaching, like the availability, the teacher-student ratio (ai is 1:1), the overall attitude towards the human/ai teacher, the lower fear of failure in front of a chatbot in comparison to in front of a teacher or a class, the costs involved (ai is cheaper than university study fees), the possibility to tailor individual learning strategies, the possibility to get an overall impression of the student performance, identifying the weight of certain aspects, the overall situational/ethical/social appropriateness etc. The field studies also reflect on the changes in performance evaluation, e.g. evaluating programming chat bots, prompting and adjusting instead of evaluating the outcome (homework, thesis, journal paper).

References
Sweller, John, “Cognitive load during problem solving: Effects on learning.” 1988
Mayer, R.E. The Past, Present, and Future of the Cognitive Theory of Multimedia Learning. Educ Psychol Rev 36, 8 (2024). https://doi.org/10.1007/s10648-023-09842-1
Jiang, D. Cognitive Load Theory and Foreign Language Listening Comprehension. Springer 2024

 

DENG Yunhua is Professor and head of the linguistics faculty, Department of English, Foreign Studies College, Hunan Normal University. Her primary research areas are cognitive linguistics, computational linguistics, and cross-linguistic comparisons.

Title: Application of Large Language Models in Language Research

Abstract: In the current field of language research, we are facing some unprecedented challenges: How to conduct interdisciplinary research? How to apply large language models to language research? Based on the BERTopic automatic extraction model, our study excavated the thematic focus of different readers towards it, conducted emotional attitude analysis to explore its overseas communication and acceptance. And another study improved the automatic recognition rate of the entities and semantic relations of “You+NP” construction. The application research based on the Large Language Model has not only promoted the understanding of the “You+NP” constructions’ semantic features and their production mechanisms, but also highlighted the significant progress of the language research paradigm transformation towards intelligence in the era of digital intelligence.

 

Mateusz-Milan Stanojević is Full Professor at University of Zagreb, Faculty of Humanities and Social Sciences

Title: Multimodal Communication in Learning, Teaching, and Research through the Cognitive Linguistic Lens: Understanding the Disruption

Abstract: Although multimodal communication is at the heart of how we communicate and learn, the multimodal component is frequently overlooked in our research. One of the reasons behind this may be its purported lack of systematicity, which inhabits sweeping generalizations, a long-standing aim of all theorizing. In this talk, I claim that our multimodal human abilities are essentially local and socioculturally situated, a view that disrupts our traditional understanding of learning, teaching, and research. I illustrate how Cognitive Linguistics helps us bridge the gap between our communicative practices and its theorizing, drawing on examples of psycholinguistic and discursive studies focusing on language use, language learning and teaching.

References

  • Frank, Roslyn M., René Dirven, Tom Ziemke, and Enrique Bernárdez, eds. 2008. Body, Language, and Mind: Sociocultural Situatedness. Berlin, New York: Mouton de Gruyter.
  • Geld, Renata, and Mateusz-Milan Stanojević. 2018. Strateško konstruiranje značenja riječju i slikom: konceptualna motivacija u ovladavanju jezikom [Strategic Meaning Construal Using Words and Images: Conceptual Motivation in Second Language Learning]. Zagreb: Srednja Europa.
  • Górska, Elżbieta. 2020. Understanding Abstract Concepts across Modes in Multimodal Discourse: A Cognitive Linguistic Approach. New York and London: Routledge.
  • Stanojević, Mateusz-Milan, Mirjana Tonković, and Anita Peti-Stantić. 2023. “When Do Metaphorical Frames Exhibit Psycholinguistic Effects? The Case of the Ego-Moving and Time-Moving Metaphor in Climate Change.” Lublin Studies in Modern Languages and Literature 47 (2): 97–107. https://doi.org/10.17951/lsmll.2023.47.2.97-107.
YANG Yuxiao is Associate Professor and director of the Linguistics Faculty, Foreign Studies College, Hunan Normal University. His research fields include second language acquisition, experimental phonetics, and psycholinguistics.

Title: The Application of Multi-modal Approaches in L2 Speech Learning in the Age of AI

Abstract: In this talk, I will present three empirical studies utilizing multi-modal approaches to the training of L2 sounds by Chinese learners of English. In the first study, visual modality of articulation was used to help learners distinguish highly similar English vowels. The training efficacy was compared with that of the machine recognition approach. The second study focuses on the tactile modality of L2 phonetic training. Rubber bands were used to enhance the articulatory distinction between English tense and lax vowels, which appeared to be more effective in comparison with the traditional repetition methods. In the third study, three gestures were employed in the training of English stresses in different intonation patterns. The gestures were developed corresponding to the three suprasegmental features of pitch, intensity and duration,enabling learners to activate both visual and proprioceptive modalities. The results of these studies suggest that the multi-modal approaches to nonnative speech learning could still harness irreplacable values under the great influence of AI, especially in the tactile and proprioceptive domains.

 

About Us

Conference Organizing Committee

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science.

Committee members: Prof. JIANG Lihua, Prof. DENG Yunhua, Prof. WAN Guangrong, Dr. YANG Yuxiao, Dr. TAN Xiaojuan, Dr. ZHANG Ying.

Hunan Normal University

Situated in Changsha, a city renowned for its historical and cultural significance, Hunan Normal University (HUNNU) is an institution of higher education designated as a national “211 Project” and “Double Top-Class Project” university. Jointly supported by the Ministry of Education and Hunan Province, HUNNU was founded in 1938 as a National Normal College (NNC), establishing it as one of the oldest normal universities in China. During the wave of university reforms in 1953, Hunan Normal College (HNC) was built upon that NNC foundation, and subsequently renamed HUNNU in 1984. In 1996, it was honored with inclusion in the “211 Project,” a prestigious initiative of the Chinese Ministry of Education to develop “100 key universities to be promoted in the 21st century.” Since 2000, HUNNU has undergone a period of significant expansion, merging with Hunan Teachers’ College, Hunan College of Politics and Law, and Hunan Medical College.

HUNNU comprises 24 colleges and offers 92 undergraduate disciplines across 11 principal categories: philosophy, economics, law, education, literature, history, science, technology, agriculture, medicine, management, and art. The university boasts six National Key Disciplines, including Ethics, English Language and Literature, Modern Chinese History, Developmental Biology, Theoretical Physics, and Basic Mathematics. Furthermore, it possesses nine Key Disciplines sponsored by the 211 Project and 22 provincial-level key disciplines designated under the 12th Five-Year Plan.

HUNNU has established partnerships with 171 universities and institutions across 41 countries and regions to foster personnel exchange and cooperation in teaching and scientific research. It has also co-established Confucius Institutes at Kazan Federal University in Russia, Wonkwang University in South Korea, and Southern Utah University in the United States.

Over its 80-year history, HUNNU has demonstrated consistent growth, even amidst the turmoil of World War II. Its faculty, across generations, have steadfastly adhered to the motto "Be humane, benevolent, excellent and diligent," working tirelessly to achieve the prosperity evident today. In recent years, driven by the “211 Project” and the “Double Top-Class Project,” HUNNU has made significant strides in discipline development, student education, faculty development, teaching research, and social service, exceeding the needs of Hunan Province in its educational, economic, and social development.

Looking forward, HUNNU embraces holistic education as its core mission, striving to become a leading comprehensive university. With distinct advantages in teacher training, it aims to achieve top-tier status in China and gain international recognition.

College of Foreign Studies, 410081 36 Lushan Rd., Yuelu District, Changsha, China

The College of Foreign Studies at Hunan Normal University traces its origins to the Department of Foreign Studies at National Normal College, founded in 1938. Its inaugural dean was Qian Zhongshu (1910-1998), a renowned scholar of Western and Chinese culture. Following Qian Zhongshu, the College benefited from the leadership of other eminent scholars, including Luo Kailan (1906-1988) and Liu Zhongde (1914-2008). Today, the College offers a first-level doctoral program in Foreign Language and Literature and hosts a research station for post-doctoral fellows. Under the leadership of Professor Jiang Hongxin, its English Language and Literature discipline has been recognized as a national key discipline. In September 2017, its Foreign Languages and Literatures discipline was admitted into the national “World First-Class Discipline Construction Project,” one of only six disciplines of its kind in China to receive this distinction.

The College comprises the Departments of English, Translation Studies, Russian, Japanese, Korean, French, and Public English. It also boasts a number of prominent research institutes, including the Hunan Center for International Cultural Communication, the Hunan Center for Sino-Russian Cultural Exchanges, the Center of American Studies, the Center of Northeast Asian Studies, the Center for Studies of British and Irish Literature, the Center of Modern Foreign Language Teaching, the Center of Cognitive Linguistic Studies, and the Center for Studies of British and American Poetry. The College publishes the Journal of Foreign Languages and Cultures, along with a corresponding Chinese journal, and supports three Confucius Institutes abroad.

The College faculty consists of 26 full professors, 44 associate professors, and numerous lecturers, of whom 51 hold doctoral degrees. The faculty includes two members of the Discipline Assessment Group under the State Council, two state-level teaching masters, and two recipients of the New Century Talent Program of the Chinese Ministry of Education. The College maintains partnerships with over 30 universities in the United States, Britain, Japan, Russia, and South Korea. Currently, it enrolls over 40 doctoral candidates, over 600 graduate students, and over 1,200 full-time undergraduates.

Adhering to the motto “international perspective, global sense, honesty, integrity, and versatility,” the College of Foreign Studies is dedicated to cultivating well-rounded and innovative talents who are both physically and mentally healthy, ethically grounded and intellectually developed, and equipped to adapt to societal changes.

Red Hen Lab™

The International Distributed Little Red Hen Lab™ is a global big data science laboratory and cooperative dedicated to research in multimodal communication. Red Hen leverages the expertise of researchers from diverse fields, ranging from artificial intelligence and statistics to linguistics and political communication, to create comprehensive datasets of parsed and intelligible multimodal communication. It also develops tools to process these data and any other data amenable to such analysis. Red Hen’s organizational structure and computational tools are designed to foster reliable and cumulative progress in the dynamic and challenging field of human multimodal communication. Understanding how humans create meaning and interpret forms necessitates this type of collaborative approach.