International Conference on Multimodal Communication: Developing New Theories and Methods

Full Conference Program | Abstracts

Find us on Twitter for the most recent conference news!

Plenary workshops

1. Mark Turner. Institute Professor and Professor of Cognitive Science, Case Western Reserve University. Co-director, the Distributed Little Red Hen Lab.  CV. markturner.org. “Red Hen tools for the study of multimodal constructions.”
2. Jakob Suchan. Ph.D. student, Human-Centered Cognitive Assistance (HCC Lab). University of Bremen, Germany. http://hcc.uni-bremen.de  | Cognitive Vision. http://www.cognitive-vision.org  —  "Computational Cognitive Vision for Human-Behaviour Interpretation"
3. Silva Ladewig, Assistant Professor, currently replacing Cornelia Müller at the European University Viadrina, Frankfurt (Oder) and Jana Bressem, academic assistant to the chair for German Linguistics, Semiotics, and Multimodal Communication at the Technische Universität Chemnitz. "Methods of Gesture Analysis—analyzing multimodality from a cognitive-linguistic perspective."
4. Peter Uhrig. Post-Doctoral researcher at the chair of English Linguistics, FAU Erlangen-Nürnberg. "Researching co-speech gesture in NewsScape – an integrated workflow for retrieval, annotation, and analysis."

Detailed Descriptions of Plenary Talks and Plenary Workshops on Methods

Plenary Speakers

Title: On establishing an integrated research agenda and workflow for multimodal communication. Abstract: Human communication is a core research area, intersecting with every other human endeavor. An improved understanding of the actual and effective complexities of human communication will have consequences for a broad range of fields, from politics and religion to education and business. Historically, the study of human communication has long been recognized as a central discipline, reaching back to antiquity when the study of rhetoric was the core element in education. Theoretical advances in the understanding of human communication date back millennia, to the early grammarians. The focus was mostly on written language, since the data record lent itself to systematic study. Human communication, however, has always been multimodal, and modern communication technologies have been developed to allow the full visual and auditory channels of face-to-face communication to be broadcast globally. These broadcasts can in turn be electronically captured and stored, making vast datasets of real world multimodal communication for the first time available for systematic scientific study. These new datasets present a radical new challenge: to develop a new and integrative model of the full complexity of human communication, building on existing advances in linguistics. To advance research into human multimodal communication and its role in human endeavors, Mark Turner and I founded The Distributed Little Red Hen Lab. Red Hen is designed to function as a global research commons, a practical and theoretical platform for research into multimodal communication. It provides core datasets, maintains a wide and rapidly growing network of researchers, develops an expanding suite of new computational tools, and facilitates the exchange of skills and the identification of the complementary forms of expertise required to make rapid progress. It aims to create an efficacious multilevel integrated research infrastructure and workflow along the following lines.

1. Introduction to multimodal communication research
2. Red Hen Primer: selective unix shell commands, applied python, how to deploy NLP engines, and the statistical package R
3. Multimodal analysis with Elan -- hands-on, best coding practices, annotations integrated into the Red Hen dataset
4. Research question development in the student's area of interest -- e.g., crossmodal constructions, complex blends, multimodal disambiguation.
5. Apply machine learning tools to annotations to create feature-specific classifiers on Red Hen servers, semi-supervised through feedback in Elan
6. Run the classifier for feature extraction and annotation on high-performance computing clusters at CWRU, UCLA, and Erlangen (cf. Audio processing pipeline)
7. Search for complex correlations between linguistic, auditory, and visual annotations in the Red Hen dataset
8. Interpretation, qualitative and quantitative/statistical analysis of the search results
9. Experimental testing of communicative effects
10. Visualization, write-up, presentations

What kind of knowledge can we expect and aspire to generate? We can generate new knowledge on the range of expression achieved and possible in multimodal communication, expending the scope of which aspects of human experience can be effectively communicated. We can generate new knowledge on the generative processes of multimodal communication, tracking patterns of repetition and innovation. We can also generate new knowledge on the epidemiology of representations, determining which forms of expressions catch on and spread most widely. We can generate new knowledge about persuasiveness and influence, examining how multimodal communication can be used effectively for truth as well as maliciously for propaganda and misinformation. Bio: Professor, Department of Communication, UCLA. Co-director of the Distributed Little Red Hen Lab, which is dedicated to the study of multimodal communication. Specialist on the development of multimodal constructions in mediated environments and on computational techniques for big data capture, transmission, and analysis. http://commstudies.ucla.edu/content/francis-steen-ph

2. Mehul Bhatt

Title: Minds, Media, Mediated Interaction: On AI and Spatial Cognition for Human Behavioral Research

Abstract: Our research at the interface of Artificial Intelligence, Spatial Cognition, and Human-Computer Interaction focusses on the design and implementation of computational cognitive systems, and mediated interaction technologies. Here, we are driven by application areas where human-centred perceptual sensemaking and interaction with cognitively founded conceptualisations of space, events, actions, and change are crucial. A recent emphasis has been on the processing and semantic interpretation of multi-modal human behaviour data, with a principal focus on highly dynamic visuo-spatial imagery.
In this talk, I will:

• make a case for the foundational significance of artificial intelligence and visuo-spatial cognition and computation research for the development of integrated analytical–empirical methods suited for the (multi-modal) study of human behaviour in diverse contexts of socio-cultural, and socio-technological significance;
• present foundational methods and assistive technologies for systematic formalisation and empirical analysis aimed at, for instance, the generation of evidence, and establishing & characterising behavioural correlates for the the synthesis of embodied cognitive experiences in select contexts.

The presentation particularly emphasises integrating analytics and empiricism for the multimodal study of human behaviour in (select) contexts: communications and media design, architecture design, and interaction design. With the support of an additional plenary tutorial, I will highlight core results and open opportunities for the semantic interpretation of human behaviour in the backdrop of “indoor wayfinding studies” and “cognitive film studies” encompassing:

• computational models of narrative:  computational cognitive vision for moving image analysis from the viewpoint of machine coding of narrative scene structure of the analysed content
• image schemas:  embodied grounding and simulation for generalised / domain-independent image-schematic analysis of narrative media
• behaviour and learning:  computational learning of human behavioural patterns vis-a-vis visuo-auditory computational narrative forms; emphasis is on acquiring qualitative or high-level knowledge from large-scale experiments / datasets
• semantic question-answering:  system-driven reasoning and query answering about digital narrative forms, and their embodied reception, e.g., for high-level attention and visual fixation analysis vis-a-vis embodied scene semantics and learnt behavioural patterns

I will showcase the manner in which semantic interpretation of human behaviour, founded on AI-based methods such as in (1–4), serves as basis to externalise explicit and inferred knowledge about embodied cognitive experiences, e.g., using modalities such as diagrammatic representations, natural language, complex (dynamic) data visualisations.

Bio: Mehul Bhatt is Professor within the Faculty of Mathematics and Informatics at the University of Bremen, Germany; and Stiftungs Professor at the German Research Center for Artificial Intelligence (DFKI Bremen). He leads the Human-Centred Cognitive Assistance Lab at the University of Bremen, Germany (HCC Lab. http://hcc.uni-bremen.de/), and is Director and co-founder of the research and consulting group DesignSpace (www.design-space.org). Mehul's research encompasses the areas of artificial intelligence (AI), cognitive science, and human-computer interaction. Of particular focus are basic topics on spatial cognition and computation, visual perception, knowledge representation and reasoning, multimodality and interaction studies, design cognition and computation, and communications & media studies (with a focus on visuo-auditory narrativity). Mehul's currently ongoing research initiatives particularly reach out with methods and technologies from ``cognition and artificial intelligence'' for integrated analytical & empirical behavioural research in psychology, social sciences, and arts & humanities. Mehul obtained a bachelor's in economics (India), master's in information technology (Australia), and a PhD in computer science (Australia). He has been a recipient of an Alexander von Humboldt Fellowship, a German Academic Exchange Service award (DAAD), and an Australian Post-graduate Award (APA). Recently steered initiatives include: CoDesign 2017: The Bremen Summer of Cognition and Design; and HCC 2016: The International School on Human-Centred Computing. http://www.mehulbhatt.org

3. Irene Mittelberg

Title: "Scenes – frames – constructions: Some ways of correlating embodied patterns of experience in language and gesture "
Abstract: Starting from the assumption that basic scenes of experience tend to underpin entrenched patterns in both language and gesture, this talk sketches the foundations of a frame-based account of discourse-integrated, communicative action. It is argued that gestures recruiting frame structures tend to metonymically profile deeply embodied, habituated aspects of scenes, e.g., the motivating context of (certain) semantic frames and grammatical constructions (e.g., Fillmore 1982; Goldberg 1998; Dancygier & Sweetser 2014). To begin with, two kinds of embodied frame structures situated at different levels of abstraction and schematicity are introduced and illustrated with multimodal discourse data (Mittelberg in press; Mittelberg & Joue 2017): i.) basic physical action frames, understood as directly experientially grounded and involving physical action and/or interaction with the material and social world; and ii.) complex, highly abstract frame structures that are more detached from the respective motivating context of experience. Shifting the focus to syntactic frames, I then turn to recent research from the Natural Media Lab aiming to contribute to the ongoing discussion on the topic of multimodal constructions (e.g., Steen & Turner 2013; Zima & Bergs 2017). Crucial questions pertain to whether, how, and under what conditions, spontaneous co-speech gestures may be said to partake in constructions or may themselves be considered constructions. First, I report on a case study showing how manual actions and interactions, such as holding or giving, may not only be said to function as blueprints for prototypical transitive or ditransitive constructions in language (Goldberg 1995), but also to underpin multimodal instantiations of impersonal existential constructions in German discourse: Spoken uses of the es gibt ‘it gives’ (there is) construction have been found to co-occur with metonymically reduced gestural enactments of holding something, thus evoking, e.g., a scene of existence, or presence, rather than a scene of object transfer (Mittelberg 2017). Finally, two pilot studies employing motion-capture data as a means to identify and analyze multimodal instantiations of frames/constructions are presented, highlighting some of the merits and challenges that come with using this technology to a) visualize movement traces; b) run similarity searches based on specific movement types; and c) derive crossmodal patterns of specific linguistic target structures and the correlating gestural practices. The overall goal is to instigate a discussion of how to combine theory-inspired, qualitative and quantitative methods to establish and better understand (and eventually predict) discourse-shaped tendencies in the crossmodal clustering of particular frames, constructions, and gestural patterns. An ensuing, more general question, being explored from different angles at this conference, is how we can derive and account for emergent regularities (Hopper 1998) or “the conventionalization of commonly used discourse patterns” (Bybee 2013: 51) within and across the manifold manifestations and genres of multimodal communication.
References:

Plenary Workshops on Methods

Title: "Researching co-speech gesture in NewsScape – an integrated workflow for retrieval, annotation, and analysis." Abstract: Finding co-speech gesture in a corpus is a time-consuming process that still involves a lot of manual work. Finding such gestures aligned to abstract grammatical constructions is even more difficult in standard corpora. This plenary workshop is designed to demonstrate a full research workflow within the NewsScape/Red Hen framework from the retrieval of the grammatical structures to thinning, classification, and the analysis of the manually annotated data using state-of-the-art tools developed within the Red Hen community. Bio: Peter Uhrig is Post-Doctoral researcher at the chair of English Linguistics, FAU Erlangen-Nürnberg. He studied English and French at FAU Erlangen-Nürnberg and Lancaster University. He defended his PhD on clausal subjects in English in 2013 and started to collaborate with the Distributed Little Red Hen Lab in 2014. His main research interests are syntax, corpus linguistics, collocation , and lexicography. His recent interest in co-speech gesture stems from his work in Construction Grammar since he hopes to be able to use co-speech gesture to find out more about the storage of grammatical constructions (i.e. which ones are more likely to be stored together and which ones are more likely to be stored separately). https://www.anglistik.phil.fau.de/staff/uhrig/