With the eMAEX-system („electronically based media analysis of expressive movement images”), we developed a multi-stage computer-based analysis routine to grasp and describe audio-visual expression patterns in their spatial and temporal dynamics. The method is based on the systematic subdivision of the audio-visual material – be it motion pictures, documentaries, news segments, advertisement clips, YouTube videos, etc – into compositional units as well as their descriptive, qualitative analysis.

In the analytical practice this results in a multi-stage process during which the audio-visual material e.g. film is segmented and analyzed on three levels:

1. Segmentation and meso-level analysis

• The film is subdivided into scenic units, that is into scenes which are characterized by a somewhat thematic as well as compositional coherency. (In research projects this ideally happens on the basis of independent subdivisions by three people, whose segmentations are merged by a fourth person.)

• Hereby, a first qualification of the scenes is already being performed: Is the scene a prologue or an epilogue? How does the scene relate thematically, scenically, and atmospherically to its preceding and its subsequent scenes? Can we find repetitive structures?

• On the basis of this division, one can already proceed with a first analysis of the scenic structure on the macro level of the film. For instance, does a specific dramaturgy emerge or can we find a classification into single acts?

• Based on this macro analysis, single or multiple scenes are identified, which are subject to a detailed micro analysis in the next step. Which scenes are affect-dramaturgical key scenes? In which scenes do scenic structures emerge which have to be investigated more intricately?

2. Segmentation and micro-level analysis

• Once scenes are appointed to a detailed analysis, they will again be temporally segmented – into units of expressive movement (EMU for short). This internal segmentation already forms the first and most important step of the detailed analysis: The scenic composition will be investigated in regard to its dynamic, spatiotemporal staging patterns which unfold as an interaction of the smaller compositional units (EMUs).

• The dynamic patterns will be described on different levels (for an extensive system which is based on these design levels, see the vocabulary of the AdA Filmontology):

  • Camera (including e.g. field size, shot sequence, camera movement and –angle, but also montage and montage rhythm)
  • Acoustics/sound design (including e.g. music, noises, speech)
  • Gestures and facial expressions (including all other aspects of acting)
  • Character constellation (including choreography of characters/objects in the sense of mise-en-scène)
  • Image composition (including distribution of contrast and color, valeurs, visual patterns, and image composition, as well as dynamics within the image)
• The identification of dominant and subdominant levels, as well as the description of their interaction within individual units of expressive movement form an effective tool.
• On the basis of this analysis, the individual units of expressive movement and the resulting dynamic scene pattern can be qualified in their affective orientation. However, the goal is not to ascribe psychologically specified, distinct emotions (joy, fear, anger, etc.). Rather, atmospheric moods and patterns of the subtle sensual conditioning of the embodied viewer’s perception are worked out.

3. Macro-level analysis

• Once the relevant scenes are described and qualified, their order defines the affect-dramaturgical structure of the film.

This basic analytical framework is applicable for various research subjects and can be adapted and further developed, depending on research interests and –perspective.

The eMAEX-system was developed in a research project focusing on the Hollywood war film;  the approach was first adapted in a project focusing on multimodal metaphoricity. A comparison of these projects shows how the three levels of analysis are focused differently: The genre research into  the Hollywood war film was systematized with an emphasis on the macro level. The starting point for the research on cinematic metaphors, however, was the dimension of expressive movement on the micro level.

The common framework is provided by the aim of developing an empirical media aesthetics. This entails the claim of a qualitatively descriptive empiricism of film analysis. The empiricism is not solely supposed to make (e.g. genre-theoretical, film-historical) theses verifiable and understandable in relation to the audiovisual material. It also offers a base for comparative analyses. Such an empirical reconstruction of movement patterns and temporal processes can lean on the deployment of video annotation software and data management. The (further) development of software for systematic annotation was initiated by the project “Audio-visual rhetorics of affect” in the context of a comparative corpus analysis.