Navigation
International Scientific Board
Center for Cognitive Science Home

 

ICMC2023: 2023 International Conference on Multimodal Communication:

AI and Multimodal Communication 

Saturday, 16 December 2023, Changsha


Copilotsforlinguists.org

Each presentation slot is 45 minutes, within which there is a period at the end of about 10 minutes for both Q & A and a break.

 

      

Hunan Normal University, Changsha, 16 & 17 December 2023

Local Venue: Lecture Hall Room 613, Foreign Studies College, Hunan Normal University
Conference directors: ZENG Yanyu and Mark Turner
会议负责人:曾艳钰教授、Mark Turner教授

There is also a conference workshop.
Time: Monday, 2023-12-18, 2:30-5pm China Standard Time.
Location: Room 611, Zhonghe Building, which is next to the College of Foreign Studies.
The workshop will not be streamed or recorded.

The 2023 Hunan Normal University International Conference on Languages and Cultures has as its theme AI AND MULTIMODAL COMMUNICATION

Conference Theme

Human beings are extraordinarily masterful at multimodal communication. Can AI be used to help study human multimodal communication? Can AI assist human multimodal communication? Can AI support human development during ontogeny through multimodal interactive communication? Recent development in AI have raised new questions and presented new opportunities. This conference is dedicated to considering the possibilities for AI involvement in the study of human multimodal communication, and for practical AI applications that engage human beings and groups of human beings through communicative channels.

Confirmed Conference Plenary Speakers

Saturday, 16 December 2023

  • 8:30-9:00 China Standard Time
    Opening ceremony.
    Mark Turner
  • 9:00–9:45 China Standard Time    
    Thomas Hoffmann. Chair of English Language and Linguistics, Catholic University, Eichstätt-Ingolstadt. Furong Scholar: Distinguished Chair Professor of Hunan Normal University.
    Title: AI and Multimodal Construction Grammar
  • 9:45-10:30 China Standard Time. (2023-12-15 3:45 pm HST. / 2023-12-15 8:45 pm EST. / 2023-12-15 10:45 pm BRT. / 2023-11-15 1:45am UTC.)
    Tiago Torrent. Professor of the Graduate Program in Linguistics, Federal University of Juiz de Fora, Brazil.
    Title: Framing the Multimodal Generative AI Debate: datasets, compositionality and the future ahead
  • 10:30-11:15 China Standard Time. (2023-12-15 4:30 pm HST. / 2023-12-15 9:30 pm EST. / 2023-12-15 11:30 pm BRT. / 2023-12-16 2:30 am UTC.)
    REN Wei. Professor, School of Foreign Languages, Beihang University, Beijing, China.
    Title: A translanguaging perspective of Chinese digital communication on social media
  • 11:15-12:00 China Standard Time. (2023-12-15 5:15 pm HST)
    Francis Steen. Associate Professor of Communication, University of California Los Angeles.
    Title: True Multimodal AI: A Prospective Assessment
  • Break 12:00-13:45 China Standard Time
  • 13:45-14:30 China Standard Time. (2023-12-16 12:45am EST)
    Mark Turner. Institute Professor and Professor of Cognitive Science, Case Western Reserve University.  Co-director, the International Distributed Little Red Hen Lab.
    Title: AI and Multimodal Frame Blending.
  • 14:30-15:15 China Standard Time.
    DENG Yunhua. Professor, Hunan Normal University.
    Title: Automatic Recognition and Feature Study of Chinese Characteristic Discourse Based on T5 Language Model
  • 15:15-16:00 China Standard Time.
    Peter Uhrig. Center for Scalable Data Analytics and Artificial Intelligence Dresden-Leipzig (ScaDS.AI) Technische Universität Dresden
    Title: Data Science Methods for Multimodal Communication Research
  • 16:00-16:15 China Standard Time.
    Closing comments.    Mark Turner. Case Western Reserve University.

    Detailed Descriptions of Plenary Talks

Thomas Hoffmann is Chair of English Language and Linguistics, Catholic University, Eichstätt-Ingolstadt. Furong Scholar: Distinguished Chair Professor of Hunan Normal University.

Title: AI and Multimodal Construction Grammar

Abstract: Multimodal Construction Grammar (Hoffmann 2017, 2021, Uhrig 2021) is an extension of Construction Grammar into the realm of multimodal communication. The present talk presents the fundamental principles of this approach (which crucially draws on conceptual blending (Turner 2014) as the cognitive processes creating multimodal utterances). After that it will explore how conversational AIs can be used as copilots (Torrent et al. 2023) for Multimodal Construction Grammar research.

  • References:
  • Hoffmann, Thomas. 2017. Multimodal Constructs – Multimodal Constructions? The Role of Constructions in the Working Memory. Linguistics Vanguard 3(1) .
  • Hoffmann, Thomas. 2021. Multimodal Construction Grammar: From multimodal constructs to multimodal constructions. In: Xu Wen and John R. Taylor, eds. The Routledge Handbook of Cognitive Linguistics. New York: Routledge, 78-92.
  • Torrent, Tiago Timponi, Thomas Hoffmann, Arthur Lorenzi Almeida, and Mark Turner. 2023 Copilots for Linguists: AI, Constructions, and Frames. (Cambridge Elements in Construction Grammar). Cambridge: Cambridge University Press.
  • Turner, Mark. 2014. The Origin of Ideas: Blending, Creativity, and the Human Spark. New York: Oxford University Press.
  • Uhrig, Peter. 2021. Large-Scale Multimodal Corpus Linguistics: The Big Data Turn. Post-doc thesis, FAU Erlangen.

     

Tiago Torrent. is Professor of the Graduate Program in Linguistics, Federal University of Juiz de Fora, Brazil.

Title: Framing the Multimodal Generative AI Debate: datasets, compositionality and the future ahead

Abstract: Multimodal Generative AI tools have been framing the recent debate on language technology. Part of the debate focuses on the impacts of such tools on downstream tasks and industry applications. Another part focuses on their risks and ethical considerations about their development, given the harmful biases found in the training data used for building generative AI tools. However, another important question to be asked is: to what extent assumptions about meaning-making present in the guidelines for building multimodal datasets used for training those tools impact and limit their performance. In this talk, I propose another framing for the multimodal generative AI debate. I argue that without a model for multimodal communication that grounds meaning in human cognition, limitations of generative AI tools will tend to persist. I support this claim with an analysis of major multimodal datasets and offer an alternative methodology for building and applying them to the development of AI tools.

 

REN Wei is a Professor of Linguistics and Vice Dean at the School of Foreign Languages, Beihang University, China. He is one of the vice presidents of the China Pragmatics Association. He also serves as an associate editor for Ampersand, a book review editor for East Asian Pragmatics, and as a board member for many journals, such as Applied Pragmatics, Contrastive Pragmatics, Discourse Context & Media, Educational Linguistics, International Journal of Bilingual Education and Bilingualism, Journal of Multilingual and Multicultural Development, Journal of Pragmatics, Language Teaching Research, and System. His research interests include pragmatics and second language acquisition. His recent publications include articles in Applied Linguistics, Applied Linguistics Review, Assessing Writing, Discourse Context & Media, International Journal of Bilingual Education and Bilingualism, International Journal of Multilingualism, Journal of Pragmatics, System and a book Second Language Pragmatics (Cambridge University Press).

Title: A translanguaging perspective of Chinese digital communication on social media

Abstract: Translanguaging has attracted much research attention in multilingual education. Many studies have demonstrated translanguaging strategies in online communication. However, less research has focused on specific speech acts and translanguaging strategies, particularly communicative practices. This talk will combine several empirical investigations, including the translanguaging strategies used by Chinese netizens in the speech act of self-praising on Weibo, Chinese youth’s attitudes toward translanguaging strategies and practices in self-praise and communication on social media, and the use of translanguaging strategies by L2 learners of Chinese on social media. In addition, the translanguaging strategies used to reflect Chinese dialects are explored. Taken together, these empirical findings aim to develop a framework for investigating the production and perception of translanguaging practices in Chinese social media communication and to provide innovative research ideas for translanguaging research on social media.

     

Francis Steen is Associate Professor of Communication at the University of California, Los Angeles.

Title: True Multimodal AI: A Prospective Assessment

Abstract : Our experience of the world is multimodal, yet early neuroscientists concluded that information is encoded digitally, in the firing or non-firing of a neuron. McCulloch and Pitts formalized this view in a threshold-based mathematical model of a neuron using binary inputs and outputs. This was the inspiration for the perceptron, the building block of deep neural networks. We now know that neurons perform far more complex kinds of information encoding and processing than envisaged by early neuroscientists. In this talk, I will explore what an updated conception of the neuron can do for artificial intelligence. Specifically, I will address how the neuron can perform the task of prospection, a key ability of human intelligence.

  • Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and brain sciences , 22(4), 577-660.
  • Li, Z., Liu, J., Zheng, M., & Xu, X. S. (2014). Encoding of both analog-and digital-like behavioral outputs by one C. elegans interneuron. Cell, 159(4), 751-765.
  • Seligman, Martin E. P., Peter Railton, Roy F. Baumeister, and Chandra Sripada (2013). Navigating Into the Future or Driven by the Past. Perspectives on Psychological Science, 8(2) 119–141
 

 

Mark Turner is Institute Professor and Professor of Cognitive Science, Case Western Reserve University;  Co-director, the International Distributed Little Red Hen Lab

Title: AI and Multimodal Frame Blending

Abstract: Frame blending is a central human cognitive process driving human creativity and conceptual innovation (Turner 2008, 2014).This presentation reviews the foundations and intricacies of conceptual frame blending, then examines possibilities for using AI to help explore multimodal frame blending. We begin with early work on frame embedding (Xi & Turner 2019) and move to the possibilities for using Generative AI Foundation models in research on frame blending. This session bridges the gap between cognitive theories of multimodal frame blending and the possibilities for AI as a research assistant in cognitive science and the study of communication.

  • References:
  • Turner, Mark. 2008. "Frame Blending." In Frames, Corpora, and Knowledge Representation, edited by Rema Rossini Favretti. Bologna: Bononia University Press. pages 13-32.
  • Turner, Mark. 2014 The Origin of Ideas: Blending, Creativity, and the Human Spark. New York: Oxford University Press.
  • Xi, Wenyue and Mark Turner. 2019. "Data Science for FrameNet and Frame Blends." Talk at the CWRU Data Science Colloquium, 24 February 2019. https://case.edu/artsci/cognitivescience/news-and-events/department-talks/data-science-colloquium/data-science-framenet-and-frame-blends

 

DENG Yunhua is currently Professor and Doctoral Supervisor of the Foreign Studies College at Hunan Normal University. She serves as the Council Member of the Cognitive Linguistics Association of China and Council Member of the Linguistics Association of Hunan Province. She worked as a Visiting Scholar at the University of Missouri (2004-2005). She won the honorary titles of "academic leader of colleges and universities" in Hunan Province and "young backbone teacher" of Hunan Province. Professor Deng specializes in linguistic theories, linguistic typology, contrastive study of English and Chinese. She has been in charge of 4 National Social Science Fund Projects. She has published 4 scholarly monographs and more than 80 theses, including 26 CSSCI journal theses, like Foreign Language Teaching and Research and Journal of Foreign Languages. She was awarded the third prize for Philosophy and Social Science Achievement in Hunan Province and the second prize for Teaching Achievement in Hunan Province.

Title: Automatic Recognition and Feature Study of Chinese Characteristic Discourse Based on T5 Language Model

Abstract: Based on the T5 language model, this study explored the named entity recognition and features of Chinese characteristic discourse. By pre-training and fine-tuning on large-scale corpora, this paper constructed a T5 language model suitable for recognizing Chinese characteristic discourse, extracting multidimensional features such as semantics, culture, and emotion to distinguish Chinese characteristic discourse from other types of texts. Experimental results demonstrate that the T5 model exhibited high accuracy in automatic recognition of Chinese characteristic discourse. Feature analysis revealed the unique expression patterns and linguistic characteristics of Chinese characteristic discourse, while discourse theory analysis elucidated the construction features of Chinese characteristic discourse. The methodology proposed in this study can be applied to the exploration of unstructured texts related to Chinese characteristic discourse, aiding in the construction of databases, knowledge graphs, and knowledge question-answering systems for Chinese characteristic discourse. This study has provided theoretical and practical implications for further research in cultural linguistics and natural language processing.

     

Peter Uhrig is Junior Research Group Leader at the Center for Scalable Data Analytics and Artificial Intelligence Dresden-Leipzig (ScaDS.AI), Technische Universität Dresden, Germany

Title: Data Science Methods for Multimodal Communication Research

Abstract: One of the major problems in research on multimodal communication is the richness and volume of the data to be analyzed. This talk will cover data collection procedures as well as data cleaning and analysis, both on the audio and on the video channels of media recordings. It will discuss the specific problems that come with such data and how they can be mitigated. Case studies will be used to show how the output of machine learning models can act as a useful input for linguistic research.

About Us

Conference Organization Committee

Directors Prof. ZENG Yanyu, Dean of the College of Foreign Studies. Mark Turner, Director of the Center for Cognitive Science. Committee members: Prof. LIU Bai, Dr. CHEN Zhongping, Dr. ZENG Jiansong, Dr. QI Xingang, Dr. QIN Yong.

Hunan Normal University

Located in Changsha, a city of great historical and cultural interest, Hunan Normal University (HUNNU) is an institution of higher education listed in the national “211 Project” and the “Double Top-Class Project” constructed jointly by the Ministry of Education and Hunan Province. Founded in 1938 as National Normal College (NNC), it is one of the oldest normal universities in China. In the wave of university reforms in 1953, Hunan Normal College (HNC) was founded on the basis of NNC. In 1984, HNC was renamed HUNNU. Gloriously, it was admitted in 1996 into the “211 Project”—one of the “100 key universities to be promoted in the 21st century” by Chinese Ministry of Education. Since 2000, it has renewed itself by merging with Hunan Teachers’ College, Hunan College of Politics and Law and Hunan Medical College in succession.

HUNNU consists of 24 colleges and runs altogether 92 undergraduate disciplines, which fall into such 11 main categories as philosophy, economics, law, education, literature, history, science, technology, agriculture, medicine, management and art. It boasts such 6 National Key Disciplines as Ethics, English Language and Literature, Modern Chinese History, Developmental Biology, Theoretical Physics, Basic Mathematics, and 9 Key Disciplines sponsored by the 211 Project, and 22 provincial-level key disciplines rated in the 12th Five-Year Plan.

HUNNU has set up partnerships with 171 universities and institutions in 41 countries and regions to push forward personnel exchange and cooperation in teaching and scientific research. It has co-established Confucius Institutes at Kazan Federal University in Russia, Wonkwang University in South Korea and Southern Utah University in the U.S.

Over the 80 years, HUNNU has been developing steadily despite the warfare of WWII. The faculty, whichever generation they were, stuck to the motto “Be humane, benevolent, excellent and diligent”, and worked hard jointly for the prosperity today. In recent years, propelled by the “211 Project” and the “Double Top-Class Project”, HUNNU has achieved much in discipline development, student education, faculty construction, teaching research and social service in the satisfaction of more than Hunan’s needs in educational, economic and social development.

While going forward, HUNNU takes holistic education as the fundamental mission, and strives to be a key comprehensive university which, with great advantages in teacher training, is top-class in China and well known abroad.

College of Foreign Studies, 410081 36 Lushan Rd., Yuelu District, Changsha, China

Foreign Studies College of Hunan Normal University dates back to Dept. of Foreign Studies of National Normal College founded in 1938. The first dean was QIAN Zhongshu (1910-1998), a famous scholar of Western and Chinese culture. After him, LUO Kailan (1906-1988), LIU Zhongde (1914-2008) and other eminent scholars worked here in succession. Now it holds the first-level doctoral program of Foreign Language and Literature and a research station for post-doctors. Under the leadership of Prof. JIANG Hongxin, its discipline of English Language and Literature was evaluated as a national key discipline. In Sept. 2017, its discipline of Foreign Languages and Literatures was admitted into the national “World First-Class Discipline Construction Project”, being one of the 6 admitted disciplines of its type in China.

It consists of Dept. of English, Dept. of Translation Studies, Dept. of Russian, Dept. of Japanese, Dept. of Korean, Dept. of French and Dept. of Public English, and boasts such institutes as Hunan Center for International Cultural Communication, Hunan Center for Sino-Russian Cultural Exchanges, Center of American Studies, Center of Northeast Asian Studies, Center for Studies of British and Irish Literature, Center of Modern Foreign Language Teaching, Center of Cognitive Linguistic Studies, Center for Studies of British and American Poetry. It publishes Journal of Foreign Languages and Cultures and a Chinese journal of the same name, and supports 3 Confucius Institutes abroad.

It has a faculty of 26 full professors, 44 assistant professors and dozens of lecturers, of whom 51 have got doctoral degrees, 2 are members of the Discipline Assessment Group under the State Council, 2 are state-level teaching masters, and 2 are awardees of the New Century Talent Program of Chinese Ministry of Education. It is a partner of over 30 universities in America, Britain, Japan, Russia and South Korea. Now it has over 40 doctoral candidates, over 600 graduate students, and over 1,200 full-time undergraduates. Adhering to the motto “international perspective, global sense, honesty, integrity and versatility”, Foreign Studies College aims to cultivate more versatile and innovative talents who are both physically and mentally healthy, both virtuous and learned and are adaptable to societal changes.

Red Hen Lab™

The International Distributed Little Red Hen Lab™ is a global big data science laboratory and cooperative for research into multimodal communication. Red Hen deploys the contributions of researchers from complementary fields, from AI and statistics to linguistics and political communication, to create rich datasets of parsed and intelligible multimodal communication and to develop tools to process these data and any other data susceptible to such analysis. Red Hen’s social organization and computational tools are designed for reliable and cumulative progress in a dynamic and extremely challenging field: the systematic understanding of the full complexity of human multimodal communication. The study of how human beings make meaning and interpret forms depends upon such collaboration.