Consistently the biggest challenge was the level of complexity for all the team members working on the film analyses. Overall the process was perceived as extremely time consuming. Following a first evaluation after several months into the project, the concepts were separated into the most relevant ones vs. the rarer ones. Team members also showed difficulties to work on such extended catalogues of concepts that were only randomly ordered by relevance. Therefore they devised ordered lists sorted according to thematic coherence, which helped finding the checkbox in a more intuitive way. Finally we ended up establishing individual layouts for each team member to take their personal focus into account. However, this approach resulted in a much more complex database architecture to collect and evaluate all the data.

Informed by the constantly evolving workflow and database architecture the crowdsourcing platform for external users [Flueckiger and Halter 2018] [Halter et al. 2019] has been developed in a modular fashion, again by Gaudenz Halter with support by Silas Weber. First, concepts are being sorted according to inner relationships, for instance positive affects related to joy vs. negative affects related to depression or aggression as elaborated and refined during the development of the workflow and following the final evaluation. Second, levels of significance of concepts and levels of complexity have been established for each area of analysis, as for instance lighting or image composition. This modular design will give users the possibility to select from a menu of concepts not only the topics they are interested in, for instance color contrasts or costume design, but also the level of complexity for each of those modules individually so that the controlled vocabulary matches best their research interest.

As mentioned above, resulting from this work was a massive quantity of data and screenshots assembled in a master analysis DB supposed to be hosted on the FileMaker server of UZH. It turned out, however, that the server was not configured for such a demanding task, which necessitated that all the screenshots were exported, down-sized and reimported what seemed like an unsurmountable task for FileMaker due to the non-standardized nomenclature for the image files. Therefore, we ultimately decided to export each screenshot into an individual folder with a defined nomenclature consisting of item ID, image ID and shot ID that were then processed externally by Gaudenz Halter and reimported automatically with a script in FileMaker. Processing included the resizing and compression of the images as well as finding the timestamps of the screenshots within the movie. Since the exported screenshots did not exactly match the content of their corresponding frame, due to resizing and compression earlier in the pipeline, the best matching frame has been determined by application of the mean squared error. This mechanism also allows to import already existing screenshots into a VIAN project and assign them to the correct locus in the video.

Similar problems occurred with the evaluation of the data. Scripts in FileMaker to assemble the data in summations for each film became too complex and the process incredibly slow and vulnerable. Even the in-house FileMaker specialists and external experts could not offer solutions. Therefore, we turned to a similar workflow to export all the data, process them externally in several Python scripts organized in a pipeline. In a first step we had to migrate the complete dataset exported from FileMaker into the VIAN WebApp database architecture. We then calculated the frequencies of keywords on a per-movie basis and related correlation matrices between keywords. This step was followed by a set of successively performed steps to enrich the existing dataset with numeric color features, including the computation of color histograms, color palettes and average color values for each segment and screenshot in a figure/ground separated manner. Finally the per-movie frequencies of keywords have been reimported into FileMaker for the evaluation. Again due to the fact that each team member received their own analysis layer organized with respect to their preferences and interests there were many inconsistencies that affected minutiae such as spelling, local and global concepts etc. Even complicated by the fact that the glossary was extended over time there were internal inconsistencies affecting the connection between the tri-partite logic of the taxonomy in the glossary consisting of classes, fields and concepts, and the organization of the values in fields of the database.

To gather consistent and significant data it is mandatory to coach users as much as possible and to illustrate concepts with precise visualizations from screenshots. The glossary database thus contains a priority field to select the most informative and clear-cut screenshots for each concept, ideally at least six screenshots from different periods for each one of them. As stated before, these screenshots are instantly available on the concepts page of the VIAN WebApp (see Figure 10), so that users get a very good idea what each keyword is referring to. These catalogues of screenshots might be extended by short video clips taken from the corpus in case where movement or other changes over time are central to the concept.

The development of VIAN has been an iterative process of development and testing with the research team. Obviously, a large number of design questions arise during such a process, especially when the number of requirements and tasks are as large as in the case of the film colors research. Clearly, there are numerous questions related to the software architecture and implementation of tools, but we have also found that developing an easy to use software can be challenging. One of the major difficulties regarding the architecture of VIAN has been to develop a software that solves the very specific need of our research project while remaining generic to be used for other projects and research topics.

Another difficulty was related to the efficient storage of the data. Most annotation tools use a human-readable file format such as XML or JSON to store the generated data permanently. These formats have the advantage that the data can easily be read even without the source tool at hand and improve interoperability. However, numeric data as generated and operated by VIAN had to be stored in a faster accessible file format. During the development we tried several approaches. We started with a simple JSON file. Once the numeric data became too large, we migrated to an SQLite database. This approach did however not scale well enough, finally we implemented a hybrid system using a human-readable JSON file for the annotations and project structure and an optimized HDF5 file for numeric data.

We have found that the most important part about the development is a short feedback loop between the developer and the users, film scholars, students or other researchers. Since there is a huge palette of statistical and analytical methods that could be implemented into VIAN. It turned out that developing in a user-centered fashion is favorable over implementing a large range of possible features. As such we tried to create a solid architectural foundation and remain generic whenever possible without introducing too much complexity.

Many video annotation software solutions have been established in the past that fulfil basic needs. But there is a big leap to developing more sophisticated tools that respond to more complex requirements. Bottom line: the devil is in the details. A very fine framework that integrates several types of players for different zoom-out functions is a powerful start to segment movies temporally, to verbally annotate these units and to extract screenshots. How well does the integrated player process diverse codecs and aspect ratios? What options does it offer to adjust segmentations and to add sub-segmentations for discontinuous entities such as parallel montage? How does it prevent overlapping segments or, by contrast, enable them? What options are there to correct existing segmentations?

Automatic temporal segmentation proved to deliver surprisingly good results that in certain instances challenged human approaches to subdivide the video stream into consistent chunks. On the negative side, auto-segmentation seemed to be much more finely grained in dark scenes and some segments were too long, especially when compared to the average lengths of segments.

To develop a robust video annotation system constant user feedback from experienced users is a necessary requirement. For the next step of auto-segmentation we envision to take music cues and sound design into account, see for instance Burghardt et al. (2016) for a very original approach to combine image and dialogue in film analysis with digital tools. Very often onset or termination of diegetic music indicate shifts in locale or time. Sound design is expressive of certain locations or temporal cues as well.

Furthermore, so-called enunciation marks such as fades to black or white, cross-fadings or intertitles should be incorporated into the system of rules for the parsing of units. Significant deviations in the resulting length of segments compared to the average should force the system to process these extremely long or short chunks again.

While we expected this task to be very demanding it turned out that — because this is one of the most important tasks in autonomous driving — deep-learning methods are currently in a very dynamic state especially with regard to extracting characters from backgrounds. YOLO [Redmon et al. 2013] was the first object recognition software applied. It provided very reliable results for the identification of humans while other objects were often misinterpreted, especially when they were partially occluded or cut off at the frame’s edges.

YOLO was combined with GrabCut [Rother et la. 2004]. GrabCut works as follows: the user initially draws a rectangle around the object to be in the foreground, GrabCut will then try to directly segment the frame, and return the result. The user can then iteratively optimize the result by marking regions that have not been identified correctly using strokes. Performing this process for each image manually would not have been feasible because of the time constraints, we thus used YOLO, an object recognition neural network to draw the initial bounding box and the strokes. However, this pipeline did not scale well enough for our purposes, demanding a large amount of resources and time. We therefore decided to use a deep-learning convolutional network to perform the pixelwise segmentation directly with semantic segmentation [Long et al. 2015] rather than the YOLO and GrabCut based approach.

A collaborative, interdisciplinary approach that connects high levels of expertise both in the domain of aesthetic analysis and computer science has proven to be mandatory for the elaboration of an analysis and visualization pipeline that respects both fields and connects them in a convincing manner. While such a statement may seem banal, in fact the actual exchange between different disciplines has been much more demanding and requires continuous adjustments from both sides. Experts from the field of humanities must be able to understand the requirements of informatics and to describe the task in a highly formalized manner. Scientists on the other hand need to be open to integrate a sense for the subtleties of aesthetic concepts to understand why minor details unexpectedly can have a significant impact on the results. The resulting pipeline should produce visualizations that respect the rigorous demands of science while also considering instant accessibility for human observers and knowledge of aesthetic distinctions at the same time.

Correlations between concepts are displayed in two ways. The features tool enables users to select the concepts from the menu. Consequently the occurrence of these concepts are then displayed over time related to the segments where they occur. Connected to the exemplary screenshots this type of visualization instantly builds the foundation to establish and test hypotheses.

The correlation between different keywords within a project or corpus-wide can be investigated using the co-occurrence matrix plots, which indicates how often every combination of keywords occurs within the scope.

In the screen video the features tool is tested with Spike Lee’s Do the Right Thing. Several typical features of this film have been selected in this visualization. In addition to the leitmotif that establishes the dominant red spectrum of the film and associates it to the topic of heat in its double sense as a temperature and as a metaphor for the rising racial tensions, the film’s aesthetics is informed by a dichotomy between the private sphere and the public space. The private sphere in interiors is often shown suffused in atmospheric diffusion again associated to the hazy damp caused by the heat in warm monochrome red tones with shafts of light filling the room, all of which are associated to a romantic tradition dating back to the early 19th century. By contrast, the aesthetics of the film’s public sphere follows a much more sober style connected to traditions of social realism with extended depth of field in wide-angle shots that show the characters in relationship to each other and to their environment. In terms of color design, the film makes use of what we call socio-political markers, culturally established conventions to denote certain social strata or official functions. For instance the protagonist played by Spike Lee himself is a pizza delivery boy and wears clothes in the colors of the Italian tricolore, white, green and red. His encounters are also defined by the central topic of race that is connected to the various ethnic groups, the Puerto Ricans, the Jamaican, the Koreans, the Italians, and the Afro-Americans, each of which is associated to different sets of hues by socio-political markers. Such a pattern of color aesthetics and meaning can easily be confirmed or further elaborated with the features tool (Figure 23).

Image plots are fantastic tools for visualizations when a researcher aims at keeping the connection to the source material. By zooming into the plots, users can look at the screenshots and see where and why a certain screenshot is present in the plot and how it is related to the film. However, because image plots’ colorimetric values are calculated based on the average of the screenshot’s color distribution, monochrome color schemes distort the visualization by being too dominant. Screenshots with more than one hue or a multitude of hues aggregate at the center of the LAB visualization or at the bottom of a Color_dT plot. Therefore, this type of visualization is best suited for early applied colors such as tinting and toning with their monochrome color distribution or for films with stark color designs in mostly one dominant hue per segment such as for instance Suspiria or Slawomir Idziak’s camerawork whose signature style often applies colored illumination and monochrome or graduated lens filters.

For many films that are not rendered well in image plots we devised an alternative solution by consecrating the full image rendition and separated the individual color values (comparison see Figure 18). That is, instead of using the average color values, we computed the color palette for a given screenshot and visualize it in the AB plane of the LAB color space. A jitter effect is applied to add some noise, making the amount of a specific hue visible within the color space. These palette dot plot visualizations now show the color schemes represented by dots for each of the colors present in a screenshot. We had to devise a method to include the spatial percentage into the dot plots. Dot plots have also become a means to show color schemes in a different way than with the typical color bars, see Figure 19. Different methods to scale and distribute colors in visualizations are offered such as zoom functions or range adjustments. Rotation is crucial for visualizations related to the L axis in the LAB space to show the distribution of hues in a meaningful way. While palette dot plots display a film’s color distribution in an intuitive way, they do not take the relative incidence into account. Therefore yet another type of visualization was introduced: heat maps that show the color distribution by means of levels of transparency corresponding to the incidence (Figure 24).

In the course of developing these visualization methods on corpus level we noticed difficulties to receive clear-cut pictures. One of the problems resulted from the fuzziness of the concepts that generated quite a high amount of noise, as elaborated in the previous section, see Section 7. However the most persistent issue that has been identified is the dominance of monochrome color schemes in the LAB visualizations, in the same fashion as in the image plots per film discussed in the previous section (see Section 7.1). Because of high levels of chromaticity in some monochrome screenshots as compared to averaging effects by variegated hues these images always stick out and therefore distort the result. This effect is even stronger in image plots that represent data on corpus level, because of the variations in different films’ color designs.

One of the first measures we took was to clean up the data. Secondly we also integrated the dot plots explained in the previous section (see Section 7) into the corpus visualizations. One of the most helpful parts of visualizations on corpus levels is the integration of the temporal segments including scene descriptions and screenshots in a sidebar next to the plots. Since the relationship between different keywords becomes complex in such a large corpus, we have implement more types of visualizations to convey the correlation and connections between concepts, the color-features associated to them and the temporal distribution over time in image plots and palette dot plots.

Differentiation between patterns and texture computationally is a non-trivial task. A naïve approach would be to assume that variance in luminance tendentially indicates a tactile quality while high variance in hue and chroma would indicate patterns. However, since patterns are not excluded from high variance in the luminance channel, this approach does not yield accurate results. Furthermore, many materials that have a tactile quality for humans often do not differ significantly numerically from flat surfaces. Co-variance of spatial frequency, color values (hue, lightness, saturation) and textures vs. patterns is tightly connected to higher order processes in human visual perception, for instance color memory and cross-modal integration, i.e. the connection of tactile experience to visual and auditory perception. As shown previously in Flueckiger’s investigation of sound design, material properties are often best detected by their acoustic cues. For a future, more elaborate system it would be an asset to include sound.

Spatial frequency and the differentiation and assessment of patterns and textures are still part of our current research. Visual complexity is one of the most important factors when it comes to style and diachronic developments. Therefore we associated an eye-tracking study to the project to gain empirical insights into the topic (see Smith / Mital 2013; Rubo / Gamer 2018). The study was conducted by Miriam Loertscher in cooperation with Bregt Lameris. For this study we chose a set of exemplary scenes for different types of image composition and complexity, for instance the clear cut type without patterns and textures as in Une femme est une femme, the type “overwhelming object world” as in Morte a Venezia with completely cluttered, layered image compositions, or Sayat Nova (The Color of Pomegranates), a film that works with many textures and material variations, often by excluding the human figure.

Results are currently being processed, but from a brief look at the heatmaps, image parts with small-scale variations detract the viewers’ gaze the most from the dominant focus on characters and most of all on faces. As Rubo and Gamer state “The influence of social stimuli and visual low-level saliency on eye movements have only recently been studied within the same datasets, and rarely in direct juxtaposition. During face perception, it was shown that facial regions diagnostic for emotional expressions received enhanced attention irrespective of their physical low-level saliency” [Rubo and Gamer 2018, 1]

The resulting hypotheses have to be tested to identify regions of high spatial frequency and by comparison with the manually gathered data to assess if these regions represent pattern or textures.