How do we understand audiovisual images? Films and other time-based audiovisual media first speak to us in a very direct manner: we perceive them. Only over the time of their unfolding, specific connections of meaning arise, which are directly connected to physically and affectively experiencing them. Tying into the research on spectator feeling and poetics of affect, the embodied understanding and thinking in audiovisual images can be put into perspective through the terms of meaning making and embodiment.

Cinematographic Spaces: Perceptual Experience and the Constitution of Meaning

How meaning emerges in cinematic images can be shown through the design of spatial perception. The concept of the image space aims at the semantic potential in the aesthetic and temporal design of images.
In Der Bildraum des Kinos: Modulationen einer ästhetischen Erfahrungsform, the terms of action space and narrative space are opposed by the term of image space. It shows that ‘viewing‘ in film doesn’t equate to an everyday ‘seeing’. More so, it is the perceptive forms of a visual construction of space themselves, which have the potential to initiate meaning formations in the act of the perceptual experience.
Image space means a dimension of cinematographic space in which all other levels of spatial representation are combined to a figuration of expression. The space is no longer defined by what happens within, but rather becomes an event itself.

Further reading:
Der Bildraum des Kinos: Modulationen einer ästhetischen Erfahrungsform. In: Umwidmungen. Architektonische und kinematographische Räume / Gertrud Koch (ed.). - First edition. Berlin: Vorwerk 8, 2005. p. 138-149.

Spectator Feeling

The essay Das Zuschauergefühl – Möglichkeiten quantitativer Medienanalyse gives an overview of the research on the affective impact of audiovisual images via the physical addressing of the spectators (see the section “The Feeling of Spectators and Poetics of Affect”). Beyond that, it explores how processes of the constitution of meaning  in audiovisual media are based on embodiment strategies and how such meaning-making-processes can be analyzed in the sense of an empirical media aesthetics. The existing media studies discourse shows that the research field is widely divided: into cultural studies-oriented media studies on the one hand, and into empirically-oriented media psychology and communication studies on the other. In relation to media reception, the term empiricism has thus far been reserved for methods of data collection in the social sciences, as well as physiological and neurophysiological measurements, and in relation to media content for methods of quantitative content analysis. It is exactly this classification which is being questioned in Das Zuschauergefühl. The spectator feeling is being understood as a dimension of continuous affect modulation, which is carried out through audiovisual design during the entire film, via the term ‘expressive movement’. Cinematic expressive movements are understood as aesthetic parameters of this continuous affect modulation. The expressivity of film can be analyzed with the help of the eMAEX-method.

In this approach, the constitution of meaning in audiovisual images is grasped as affectively based. With numerous concepts on ‘thinking in cinematographic images’ - from Münsterberg and Eisenstein to Deleuze, Sobchack and Rancière - film studies offers theoretical models that describe an intertwining of perceptive, affective and cognitive processes. In this regard, understanding in film-viewing is based temporally on the spectator’s dynamically affected experience.

Further reading:
Das Zuschauergefühl – Möglichkeiten qualitativer Medienanalyse. (By Hermann Kappelhoff and Jan-Hendrik Bakels). In: Zeitschrift für Medienwissenschaft, 5 (2), 2011. p.78-96.
(Online: http://www.zfmedienwissenschaft.de/index.php?HeftBeitragID=79, accessed: 23.04.2012)

Multimodal Metaphor

A concept of multimodal metaphor, which views embodiment and the constitution of meaning in direct relation with one another, was developed in the interdisciplinary project “Multimodal Metaphor and Expressive Movement” at the cluster of excellence “Languages of Emotion” from 2009 to 2013 under the direction of Cornelia Müller (Linguistics, Europa-Universität Viadrina) and Hermann Kappelhoff (Film Studies, Freie Universität Berlin). An approach was developed which grasps multimodal metaphors as temporal meaning making processes. Recordings of everyday conversations were investigated, as well as TV-news, classical Hollywood films and contemporary German films. The project’s core idea is that the embodiment principles of gesture, voice, and facial expressions in everyday communication are comparable with those that occur during the reception of audiovisual images. From this perspective, the aesthetic design of a film or video clip is an essential part of the emergence of meaning.

The following dissertations were written in the course of this project:
Dorothea Horst (Linguistics): Meaning Making and Political Campaign Advertising. A Cognitive-Linguistic and Film-Analytical Perspective on Audiovisual Figurativity (De Gruyter 2018).
Sarah Greifenstein (Film Studies): Tempi der Bewegung – Modi des Gefühls. Expressivität, heitere Affekte und die Screwball Comedy (De Gruyter 2020).
Christina Schmitt (Film Studies): Wahrnehmen, fühlen, verstehen: Metaphorisieren und audiovisuelle Bilder (De Gruyter 2020).

Metaphoric Meaning Making

The essay Expressive movement and metaphoric meaning making in audio-visual media summarizes the result of this research. This model’s basic concept can be grasped as follows:
In everyday conversations, it is not only abstract information that is being exchanged. More so, the whole body is a part of the conversation situation: facial expressions and gestures are an essential part of the communicative relation and give the opposite person an idea of what we think and feel. The same principle can also be found in the reception of audiovisual media: in films, moods and meanings articulate themselves – similar to gestures in a conversation – via sensorial qualities and movement patterns. The multimodal interaction of filmic means such as camera movement, mise-en-scène, image composition, acting, sound, and montage creates meanings as concrete sensual and affective experiences in the spectator’s body. The metaphor is always to be viewed as a progress, which seeks to grasp the act of viewing on the level of experience and understanding in the sense of „understanding and experiencing one kind of thing in terms of another“ (Lakoff & Johnson).

The metaphoric content of meaning becomes tangible as an embodied, dynamic process through the aesthetic staging. Experiencing as an affective process is what establishes understanding in the first place. For example, in a news report (Tagesschau) during the 2008 financial crisis, the German economy is metaphorically experienced as a ship. Not only are the domains of ship and economy unified in the audiovisual metaphor. More so, this entails a physically affective experience that the ship (and thus also the economy) is difficult to steer and keep afloat. The metaphors are to be understood as a physically sensual execution, as a concrete experience.

Further reading:
Schmitt, Christina, Sarah Greifenstein and Hermann Kappelhoff (2014) Expressive movement and metaphoric meaning making in audio-visual media. In: Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill and Jana Bressem (eds.), Body _ Language_ Communication: An International Handbook on Multimodality in Human Interaction Vol. 2. (Handbooks of Linguistics and Communication Science 38.2.), Berlin/Boston: De Gruyter Mouton.

Embodied Meaning Making

The essay Embodied meaning construction. Multimodal metaphor and expressive movement in speech, gesture, and feature film (together with Cornelia Müller) shows how the cooperation between linguistics and film studies has led to a comparability of movement patterns in face-to-face communication and audiovisual media.

The embodied emergence of meaning through multimodal metaphors in everyday communication, as well as in audiovisual media, is based on concrete sensory experiences of cinematic or physical-gestural expressive movements (see sections Spectator Feeling and Poetics of Affect). The activation of multimodal metaphors emerges only through the temporal interaction of various metaphoric statements, be it hand gestures, which accompany a verbal statement, or various gestural patterns of movement in films. Such temporal and multimodal processes in everyday conversations and in film can be viewed as a basic form of embodied meaning construction.

Further reading:
Embodied meaning construction. Multimodal metaphor and expressive movement in speech, gesture, and feature film. (By Hermann Kappelhoff and Cornelia Müller). In: Metaphor and the Social World, 1 (2), 2011. p. 121-153

"For the spectator, the opposition between the white society and the character Julie is not graspable at the level of the action, but at the level of the expressive movement units that the spectators experience. Figure 13 presents a schematic overview of our analysis. It shows how the expressive movement units analyzed above evolve and change over time, and how thereby one leading multimodal metaphor emerges that will become further elaborated over the unfolding time of the film: The society is experienced and understood as a static structure with a dynamic force inside it."

Timing and Embodiment in Melodrama and Comedy

By comparing two scenes from MAGNIFICENT OBSESSION and ADAM'S RIB in Feeling Gloomy or Riding High: Timings of Melodrama and Comedy (together with Sarah Greifenstein), the temporalities of the melodrama and the comedy are juxtaposed. In addition to the different ways in which melodrama and comedy affect their audiences, another focus lies on the different strategies of embodiment that are at work in these two modes.

„The two forms of staging in melodrama and comedy can be linked back to the corporal forms of expression of laughing and crying in their temporal shaping, or described in temporal congruence with them. […] Both phenomena emerge from the fact of being confronted with something that one no longer knows how to react to, and as a consequence the body itself responds to the situation. The cinema produces multiple forms of affectedness. What the visual and perceptual modulations of cheerfulness and sadness, emotion and sensitivity have in common is that they are especially intensive forms of addressing spectators in their embodiment, of pushing them to the limits.“

The eMAEX-System for the Analysis of Cinematic Expressivity

The eMAEX-System makes it possible to empirically-descriptively grasp the expressive dimension of audiovisual images in their temporal unfolding, which means describing the aesthetic design of audiovisual moving images specifically in regards to how they make the spectators experience specific emotions through physical address.
This method of analysis is specifically aimed at not only analytically grasping singular films, but also groups of films and genres as a whole, by working together. In order to compare films and their affective strategies with one another, films are analyzed on three different levels, the individual segments are described in their interaction and organized in online databases. The corresponding film clips can be found directly next to the analysis texts, so that it is possible to work directly with the research object.
This is linked to a series of different visualization and categorization procedures that enable analysts to compare individual segments or films in their entirety with one another and thus recognize recurring patterns or deviations, document them, and make them accessible to other researchers.

The eMAEX analysis method, which aims at the differentiating description of cinematic dramaturgies of affect (see also Spectator Feeling and Poetics of Affect), was developed based on the classic war film genre (see research focus Genre, Sense of Community, and History). It is now used in different areas and forms the basis for analyzing multimodal metaphors (see above) in audiovisual media amongst other things.

The existing data has also been transferred to a MediaWiki-platform, in order to be able to use a wider range of functions, such as the variable tagging of individual segments of analysis.