Disruption in Education: Multimodal Communication in Education, Research, and Learning 

There is also a workshop: Disruption in Practice

Conference: Saturday, 14 December 2024.

Plenary Speakers

• 8:30-9:00 China Standard Time. Moderator: JIANG Lihua (Sat, 14 Dec 2024 at 00:30 UTC; Fri, 13 Dec 2024 at 19:30 EST; Sat, 14 Dec 2024 at 01:30 CET; Sat, 14 Dec 2024 at 00:30 GMT; Fri, 13 Dec 2024 at 16:30 PST.)
  • Group Photo at Tenglong Building
  • Introduction by Professor JIANG Hongxin, Chair of the University Council
  • Introduction to the Theme of ICMC2024 by Mark Turner
• 9:00–9:45 China Standard Time. Moderator: WAN Guangrong    
  Michael Schoop. Senior Vice-President, Talent, Greater Cleveland Partnership. Previously Vice-President, Cuyahoga Community College, President of the Metro Campus of Cuyahoga Community College.
  Title: Disruption and Realignment in U.S. Higher Education in the New World of Work
• 9:45-10:30 China Standard Time. (Fri, Dec 13, 2024 at 8:45 pm EST; Fri, Dec 13, 2024 at 10:45 pm BRT; Sat, Dec 14, 2024 at 1:45 am UTC.) Moderator: WAN Guangrong
  Tiago Torrent. Professor of the Graduate Program in Linguistics and the head of the FrameNet Brasil Lab at Federal University of Juiz de Fora, Brazil. Research Productivity Grantee of the Brazilian National Research Council for Scientific and Technological Development. Professor of the Graduate Program in Linguistics, Federal University of Juiz de Fora, Brazil.
  Title: Reframing Education in the Age of Copilots: multimodal language capacities for an ever-changing AI-oriented world
• 10:30-10:40 China Standard Time. Coffee & tea break.
• 10:40-11:25 China Standard Time. Moderator: TANG Yanling
  CHEN Fei Professor, School of Foreign Languages, Hunan University.
  Title: The Influence of Audiovisual Cross-Modal Integration on Speech Perception and its Implications for Speech Training
• 11:25-12:10 China Standard Time. Moderator: TANG Yanling
  Renata Geld. Founding Director, Center for Cognitive Science, Associate Professor, Co-head MA in AppCogSci, cognitive science, cognitive linguistics, language education, interdisciplinarity in HE, creativity and creative ecosystems.
  Title: Learning as an inherently multimodal process: cognitive science of education
• Lunch Break 12:10-13:45 China Standard Time
• 13:45-14:00 China Standard Time. (Sat, 14 Dec 2024 at 05:45 UTC; Sat, 14 Dec 2024 at 00:45 EST; Sat, 14 Dec 2024 at 06:45 CET; Sat, 14 Dec 2024 at 05:45 GMT; Fri, 13 Dec 2024 at 21:45 PST.) Moderator: CHEN Minzhe
  Mark Turner. Institute Professor and Professor of Cognitive Science, Case Western Reserve University.  Co-director, the International Distributed Little Red Hen Lab.
  Title: Collaborating with Synthetic Agents
• 14:00-14:30 China Standard Time. Moderator: CHEN Minzhe
  Martin Woesler "Xiaoxiang Scholar" Distinguished Professor and Director of the Jean Monnet Research Centre of Excellence on the Digitalization of University Teaching, Foreign Studies College, Hunan Normal University. Research fields include use of ai in foreign language acquisition, cognitive load theory, appropriateness theory, comparison of ai/human teachers, field work to identify the role of different modals/senses, e.g. by field work with learners with disabilities.
  Title: Multimodal Learning from the perspective of the cognitive load theory in the age of AI
• 14:30-15:15 China Standard Time. Moderator: CHEN Minzhe
  DENG Yunhua Professor, Foreign Studies College of Hunan Normal University. Head of the linguistics team in the English department.
  Title: Application of Large Language Models in Language Research
• 15:15-15:25 China Standard Time. Coffee & tea break.
• 15:25-16:10 China Standard Time. Moderator: TAN Xiaojuan
  Mateusz-Milan Stanojević. Full Professor at University of Zagreb, Faculty of Humanities and Social Sciences
  Title: Multimodal Communication in Learning, Teaching, and Research through the Cognitive Linguistic Lens: Understanding the Disruption
  
• 16:10-16:55 China Standard Time. Moderator: ZHANG Ying
  YANG Yuxiao Associate Professor and Director of the Linguistics Faculty, Foreign Studies College, Hunan Normal University.
  Title: The Application of Multi-modal Approaches in L2 Speech Learning in the Age of AI
  
• 16:55-17:10 China Standard Time. Moderator: WANG Yuqing
  Concluding remarks.    Mark Turner. Case Western Reserve University.

Title: Disruption and Realignment in U.S. Higher Education in the New World of Work

Abstract: Demographics and economics are remaking U.S. higher education. Declining enrollment, rising costs, and questions about the relevance of traditional degrees are reshaping the landscape. At the same time, the U.S. economy is evolving rapidly, with businesses facing a growing skills gap and labor shortages, particularly in critical sectors like technology, healthcare, and manufacturing. The key question: how can higher education and business work together to bridge this gap?
Higher order skills are essential to the growth of modern economies—especially in the age of synthetic intelligence. As the world of work is transformed, the workforce must constantly reskill and upskill. How can universities provide flexible, industry-aligned programs to support this ongoing learning? What role should businesses play in fostering continuous education for their employees?
This talk explores the disruption in a market for U.S. higher education divided between prestige and economic mobility. It highlights the "last mile" problem-the gap between credentials and learning that has economic value-caused by fragmented systems and misaligned stakeholders. The challenge: how might we use data and artificial intelligence to we ensure that higher education is better connected with economic demands and meets the needs of the U.S. workforce?

Title: Reframing Education in the Age of Copilots: multimodal language capacities for an ever-changing AI-oriented world

Abstract: Schools are intended to guide pupils in their transition from family life to social and work life. Throughout the years, schools and universities have – not without some delay – resonated the changing needs of society, conforming their methodologies and approaches to education to match the expected abilities their alumni should have to be successful in the world they would encounter after graduating. The current pace of technological innovation has been challenging one foundational assumption of teachers and professors: that the world their students will find when leaving schools and universities will be the same they experienced when they started their education. In this talk I revisit language capacities that have been populating school and university curricula for the past century and discuss the extent to which the Age of Copilots imposes changes to such capacities. In particular, I address the growing importance of collaborative creative and editorial practices, highlighting the potential of synthetic agents to augment students' proficiency in multimodal communication practices.

Title: The Influence of Audiovisual Cross-Modal Integration on Speech Perception and its Implications for Speech Training

Abstract: This report examines the influence of audiovisual cross-modal integration on speech perception and its implications for speech training. The first section explores the effects of audiovisual integration on speech processing, focusing on stimulus factors (such as the impact of visual cues like facial expressions and gestures, as well as speech rate on Mandarin tone perception) and listener factors (including cross-modal speech processing mechanisms in individuals with hearing impairments using cochlear implants or hearing aids). The second section investigates the role of audiovisual integration in speech training, covering areas such as second-language pronunciation training and speech rehabilitation. Through these studies, we delve into the potential of audiovisual integration in enhancing speech perception and training, offering new perspectives for the development of speech learning and instructional methodologies.  

Title: Learning as an inherently multimodal process: cognitive science of education
Abstract :Students, irrespective of age, learn best when they can engage with information through multiple formats, enhancing understanding, retention, and the ability to apply knowledge. Multimodal learning is far richer than merely presenting information in different sensory formats; it taps into the interconnected and dynamic processes of the mind, where ideas are synthesized, visualized, felt, and embodied. Learning is an active, imaginative, and deeply personalized process. In other words, learning is inherently multimodal. Real learning involves the construction of mental connections, the visualization of concepts, and personal engagement with material that is both emotional and embodied. These cognitive processes enable learners to relate to new ideas, imagine applications, and draw on prior knowledge to create networks of meaning. While learning through multiple modes is natural in the real world, classroom environments often fall short in providing these rich, varied experiences. To foster deeper, more meaningful engagement, educational approaches must shift toward more authentic, multimodal learning experiences.
In modern education, authenticity and experience must become imperatives. Learning experiences should encompass hands-on, real-world tasks as well as technologically driven contexts, contributing to a dynamic form of experiential learning. Authentic learning environments, whether grounded in real-world engagement or enhanced by immersive, technology-driven simulations, promote active participation, reflection, and the meaningful application of knowledge. Furthermore, to fully prepare students for the complexities of the 21st century, it is increasingly critical to integrate the humanities and social sciences with STEM, cultivating a multidisciplinary approach that reflects real-world challenges. By integrating these diverse contexts and perspectives, we equip students with the adaptability, insight, and holistic understanding necessary to navigate a rapidly changing world.
In this talk, I will illustrate this approach with examples from a new graduate program in Applied Cognitive Science that was designed and accredited at the University of Zagreb. This program addresses the urgent need to integrate the humanities and social sciences with technical fields and STEM, providing students with a well-rounded, interdisciplinary foundation. Structured around the principles of multimodal and experiential learning, the program immerses students in real-world challenges and technology-enhanced environments to foster deep cognitive engagement. I will discuss how the program was developed, emphasizing key aspects of its curriculum, which combines hands-on projects, interdisciplinary collaboration, and immersive simulations. Together, these elements equip students with the skills and adaptability needed to thrive in today’s complex educational and professional landscapes.  

Title: Collaborating with Synthetic Agents

Abstract: Throughout Education, Research, and Learning, we now collaborate and communicate with synthetic agents—AI, robots, computational systems. These synthetic agents can use full multimodal communication. Communication depends on common ground. How can we establish common ground with synthetic agents? How can synthetic agents be used as part of teams, in distributed fashion, across networks? How can we trust synthetic agents? What is the future of multimodal communication in a university filled with synthetic agents?

Title: Multimodal Learning from the perspective of the cognitive load theory in the age of AI

Abstract: This paper introduces some theoretical background of ongoing field studies on ai and human teaching at European and Chinese universities, including virtual learning environments. According to dual-coding theory, humans process visual and auditory information separately (at the initial stage) in visual channel (eyes) and auditory channel (ears) and integrate the resources subsequently. If working memory resources can be allocated to these two channels, the working memory load can be spread or shared, resulting in lower cognitive load (Sweller 1988) experienced by learners. In general, a more holistic learning triggering several senses like video, audio, touch, smell, and senses for space, time etc. and working memory resources can be allocated to several channels, the working memory can be spread, resulting in lower cognitive load experienced by learners. Not only the cognitive load is lower, also the remembering and associating of the different sensual/modal remembrances into the big picture of the stuff learned ("integration") goes far easier if several different impressions contribute. Therefore, a reduction of channels like in on-screen teaching during the pandemic represents a higher cognitive work load and this can be equalized by enhancing virtual learning environments to match real-life student-teacher situations with their diverse stimuli for different channels. Accordingly, learning with an ai chatbot becomes more effective when the chatbot is humanized (antropomorphized) and when it stimulates diverse senses (modals). This is one of the hypotheses we try to proof with our field research carried out at different universities in Europe and China. However, we are also interested in identifying other factors playing into the success or failure of ai teaching vs. human teaching, like the availability, the teacher-student ratio (ai is 1:1), the overall attitude towards the human/ai teacher, the lower fear of failure in front of a chatbot in comparison to in front of a teacher or a class, the costs involved (ai is cheaper than university study fees), the possibility to tailor individual learning strategies, the possibility to get an overall impression of the student performance, identifying the weight of certain aspects, the overall situational/ethical/social appropriateness etc. The field studies also reflect on the changes in performance evaluation, e.g. evaluating programming chat bots, prompting and adjusting instead of evaluating the outcome (homework, thesis, journal paper).

References
Sweller, John, “Cognitive load during problem solving: Effects on learning.” 1988
Mayer, R.E. The Past, Present, and Future of the Cognitive Theory of Multimedia Learning. Educ Psychol Rev 36, 8 (2024). https://doi.org/10.1007/s10648-023-09842-1
Jiang, D. Cognitive Load Theory and Foreign Language Listening Comprehension. Springer 2024

Title: Application of Large Language Models in Language Research

Abstract: In the current field of language research, we are facing some unprecedented challenges: How to conduct interdisciplinary research? How to apply large language models to language research? Based on the BERTopic automatic extraction model, our study excavated the thematic focus of different readers towards it, conducted emotional attitude analysis to explore its overseas communication and acceptance. And another study improved the automatic recognition rate of the entities and semantic relations of “You+NP” construction. The application research based on the Large Language Model has not only promoted the understanding of the “You+NP” constructions’ semantic features and their production mechanisms, but also highlighted the significant progress of the language research paradigm transformation towards intelligence in the era of digital intelligence.

Title: Multimodal Communication in Learning, Teaching, and Research through the Cognitive Linguistic Lens: Understanding the Disruption

Abstract: Although multimodal communication is at the heart of how we communicate and learn, the multimodal component is frequently overlooked in our research. One of the reasons behind this may be its purported lack of systematicity, which inhabits sweeping generalizations, a long-standing aim of all theorizing. In this talk, I claim that our multimodal human abilities are essentially local and socioculturally situated, a view that disrupts our traditional understanding of learning, teaching, and research. I illustrate how Cognitive Linguistics helps us bridge the gap between our communicative practices and its theorizing, drawing on examples of psycholinguistic and discursive studies focusing on language use, language learning and teaching.

References

• Frank, Roslyn M., René Dirven, Tom Ziemke, and Enrique Bernárdez, eds. 2008. Body, Language, and Mind: Sociocultural Situatedness. Berlin, New York: Mouton de Gruyter.
• Geld, Renata, and Mateusz-Milan Stanojević. 2018. Strateško konstruiranje značenja riječju i slikom: konceptualna motivacija u ovladavanju jezikom [Strategic Meaning Construal Using Words and Images: Conceptual Motivation in Second Language Learning]. Zagreb: Srednja Europa.
• Górska, Elżbieta. 2020. Understanding Abstract Concepts across Modes in Multimodal Discourse: A Cognitive Linguistic Approach. New York and London: Routledge.
• Stanojević, Mateusz-Milan, Mirjana Tonković, and Anita Peti-Stantić. 2023. “When Do Metaphorical Frames Exhibit Psycholinguistic Effects? The Case of the Ego-Moving and Time-Moving Metaphor in Climate Change.” Lublin Studies in Modern Languages and Literature 47 (2): 97–107. https://doi.org/10.17951/lsmll.2023.47.2.97-107.

Title: The Application of Multi-modal Approaches in L2 Speech Learning in the Age of AI

Abstract: In this talk, I will present three empirical studies utilizing multi-modal approaches to the training of L2 sounds by Chinese learners of English. In the first study, visual modality of articulation was used to help learners distinguish highly similar English vowels. The training efficacy was compared with that of the machine recognition approach. The second study focuses on the tactile modality of L2 phonetic training. Rubber bands were used to enhance the articulatory distinction between English tense and lax vowels, which appeared to be more effective in comparison with the traditional repetition methods. In the third study, three gestures were employed in the training of English stresses in different intonation patterns. The gestures were developed corresponding to the three suprasegmental features of pitch, intensity and duration，enabling learners to activate both visual and proprioceptive modalities. The results of these studies suggest that the multi-modal approaches to nonnative speech learning could still harness irreplacable values under the great influence of AI, especially in the tactile and proprioceptive domains.