Leonardo, Morelli, and the Computational Mirror

Giovanni Morelli (1816-1891) was one of the foremost connoisseurs — masters of art attribution — in the history of art, and the method he proposed for identifying and authenticating old master pictures has had enormous impact on the discipline [Agosti et al. 1993] [Hinojosa 2009] [Caglioti et. al. 2018] [Barni 2016]. Morelli’s career trajectory spanned the fields of medicine, politics, and art criticism, and could perhaps have only been fostered in the bubbling cauldron that was the European nineteenth century [Anderson 2020]. Born in Italy to parents of Swiss origin, Morelli moved to Munich at age 17 to study medicine at the Ludwig Maximilian University. Although Morelli never practiced medicine, his training in that field would prove foundational for the method of art attribution that he developed later in his career. After graduation, Morelli traveled throughout Europe fostering his love of art and a network of associations in the European art market. In 1848 he was swept up in the revolutionary fervor that was overtaking the Continent and, as a consequence, began a long and somewhat complicated involvement in the emergence of the Italian Republic. He was a senator in the early Republic and served on the committee that established the system of Italian national museums that is largely still intact, albeit in a modified form.

It was only in 1874, at the age of 58, that Morelli began publishing the rudiments of his method for attributing old master paintings. From that moment, until his death in 1891, Morelli wrote numerous volumes, which quickly gained a wide audience. His interventions were published in German under a double-pseudonym: the writings purported to be the work of a Russian man named Ivan Lermolieff, which were then translated into German by Johannes Schwartze, itself a translation of the Italian name “Giovanni Morelli” into German. These publications took the form of dialogues in which Lermolieff converses with an anonymous Italian aristocrat. What we now think of as the “Morellian Method” for attributing artworks emerges as the Italian helps Lermolieff see paintings in a new way. Importantly, the “Morellian method” was not laid down in normative terms; rather, it emerges as the product of dialogic exchange that allowed Morelli to give voice to the discomforts provoked by his new method. The Lermolieff/Morelli figure plays the role of the rube: he is a foreigner who has come to Italy in search of great art. The dialogue’s other character serves as his educated guide, introducing this revolutionary new mode of connoisseurial analysis based on the revealing detail.

The rudiments of Morelli’s method are familiar to art historians operating in all European language traditions [Davis 2011, 86–96]. Prior to his work, the attribution of paintings was a chaotic enterprise entangled in questions of nationalism, capitalism, prestige, and the decline of European aristocracy. Generally speaking, claims to authorship and authenticity were made on the basis of a painting as an integral whole: every piece of the painting was believed to work together to reveal the identity of the artistic agent responsible for having made the picture. This approach is predicated on what Morelli described rather derisively as the “general impression” of a picture, and he developed his own method consciously in opposition to this approach [Morelli 1893, 22–3].

Morelli’s wager was that individual details were key, not this “general impression.” His method assumed that painters revealed their true identity in the small places where they — and by extension also the beholders — were paying the least attention. Anatomical features like fingernails, earlobes, toes, and noses became the locus of authorial identity. Morelli argued that painters were consistent in the way that they painted these features throughout their careers and that these elements became something akin to a personalized pattern. He came to describe these patterns as Grundformen, or “ground forms,” and he offered line-drawn examples in his publications (see Figure 2).[3]

In his 1890 (English trans., 1893) dialogue on the Italian paintings in the Borghese and Pamphili collections, Morelli offers an excellent example of how to use Grundformen to make an argument about the attribution and dating of an artwork. In this passage, Morelli stages a scene in which the anonymous Italian companion outlines his method quite clearly for Lermolieff while they are standing in the presence of a portrait of Bishop Ludovico Beccadelli, signed and dated by Titian in 1552. The solidity of this attribution allows Morelli to use the painting as a tool for recognizing the Grundformen that truly distinguish Titian’s art and might thereby be used to solidify other attributions:

It is my object to make you notice everything in a work of art, and in time you will come to see that even details, in themselves insignificant, may lead us to the truth… Look at the hand in this portrait, particularly at the ball of the thumb, which is too strongly developed and at the round form of the ear. In all his early works, and in most of those in his middle period till between 1540-1550, Titian adheres to the same round form of ear... This peculiarity in the ball of the thumb also frequently occurs in his other paintings and in his drawings.  [Morelli 1893, 46–7]

This discussion is illustrated by a small schematic line drawing of a hand with the caption “Tizian's Daumenballen,” which is rendered in the English translation as “The Ball of the Thumb in Titian’s Works” (see Figures 3a & 3b) [Morelli 1893, 47].[4]

That Morelli’s texts illustrate the Grundformen through line drawings and not full illustrations that more faithfully reproduce the actual image-in-the-world under scrutiny was not a technical limitation of the day. By the time that Morelli was publishing his books, it was possible to mass produce art historical volumes with full page illustrations of original works of art using a number of techniques, including lithography, engraving, and etching. His choice emphasizes his use of abstraction as a central technique and in so doing also reveals something disconcerting in Morelli’s method, which is that it is predicated on what we might think of as a kind of aesthetic dissection of the painting itself, a shredding of its holistic integrity and a disregard for the particulars of pictorial representation as instantiated in a given painting.

Indeed, the reduction of the entire work of art to the telling detail is one of the more charged elements of Morelli’s method, and he recognized it as such. Rather than see full-page illustrations of the paintings, the reader, like Lermolieff, is asked to encounter an odd array of noses, earlobes, and fingers organized as a grid to facilitate the process of differentiation between artists. With this approach, Morelli’s method challenged the sensibilities of what was, at that point, traditional art history, and the Lermolieff/Morelli character expressed repulsion at the aesthetic violence to which the painting is subjected by his Italian companion, as it is subdivided into hands, ears, and mouths, bemoaning that the “general impression” of the overall work of art is declassed to an afterthought [Morelli 1893, 19–22].

The Grundformen isolated features of the artwork and was a disorientating approach at the time, but for Morelli, the cost was worth the benefit. These schematizations facilitated attribution by staging comparisons, and this step-wise workflow in part helps explain why Morelli’s method is almost irresistible to those interested in algorithmic logic.[5] Morelli’s method was to reduce paintings to a limited set of discrete patterns, and in so doing, locate proxies for authorship in meaningful details he believed had been unconsciously embedded in the pictures. The effectiveness of the Grundformen could not be revealed by a casual look at the entire picture, but instead became potent only through their isolation, extraction, and comparison.

In the same way that the results of Big Data analyses are at their most thrilling when they arise from patterns that are not obvious during data collection, Morelli also strove to reveal invisible patterns of reality that are actually hidden in plain sight.[6] The connection to Big Data here is not simply coincidental. In the twenty-first century, Morelli’s approach might read to the astute technologist like a description, although imprecise, of a set of functions that, once formalized, could be effectively and successfully performed by a digital computer. For example, a supervised system of digital Morellian classification might begin with the identification of Grundformen, that is, the image-based features that typify a particular artist. It might then computationally select and extract those detail-features from a dataset of digital images. Next, it might identify a procedure that would best group these details, and classify them into artist-based clusters. With this trained system in hand, Grundformen could then be extracted from unattributed paintings. By comparing these unattributed features to the computationally classified set, one could produce a probabilistic attribution of this unattributed work of art to an artist, or distribute that probability over a set of artists, all by directly implementing Morelli’s assumption that similarities in the appearance of the Grundformen are effective proxies for an artist’s stylistic “hand.”[7]

Both this brief sketch of a computational Morellian system — as well as Morelli’s original procedure — are fundamentally designed to answer one critical question, “Who painted this painting?” But, as we aim to show in this paper, the notion that art attribution is a “problem” that can be “solved” or a question that can be “answered once and for all,” misses a critical point. Art attribution is a practice fully embedded in sociality. Indeed, as contemporary art historians, we have found Morelli's method to be most useful not to answer the question, “Who painted this?” but rather as a tool for better understanding all the complex, human-facing components that make up the question itself.

Our project is not invested in attempting to operationalize Morelli’s system, in part because previous scholars have already shown that this is indeed already possible, if one proceeds with the assumption that art attribution is a problem to be solved. Adrian J. Ryan, in a 2009 dissertation produced for the Department of Classics at the University of KwaZulu-Natal in South Africa, produced just such a system, and we have found that his research offers a fruitful case study for comparison. Ryan is a computer scientist/technologist by training, who, in returning to school for his degree in Classics, created a formalized, digital system designed to imitate and reproduce the Attic vase-painting attributions of Sir John Beazley. We have found his work especially useful for revealing a number of differences between a digital-art-historical understanding of the utility of Morellian method and a computer-scientific one. Most critically, Ryan’s work highlights what it looks like when Morelli’s methods are used to “look for answers,” rather than to situate the work of attribution within a social, art-historical context.

The guiding star for Ryan’s computational system of art attribution is not Giovanni Morelli himself, but instead one of the most important scholars in the history of the humanities to have implemented a Morellian Method: Sir John Beazley (1885-1970). Beazley has long served as a disciplinary figurehead for Morelli’s style of artistic attribution, not only in his home field of Attic vase-painting, but also beyond [Davis 2011, 86–96] [Driscoll 2019] [Neer 2005] [Smith 2005]. Ryan states that his overall goal in implementing a computerized system of art attribution is to discover whether or not, “a computer may be taught to attribute in the same way that an art historian can” [Ryan 2009, 56]. He is clear, both in this passage and throughout his study, that his aim is not to design a classifier that can attribute paintings in some natively computational way, or to justify “the practice of connoisseurship by means of statistics” [Ryan 2009, 14], but instead, “to prove or disprove...that machines may be taught to attribute in the same way [as a human expert]” [Ryan 2009, 2]. The system Ryan would come to produce is, for all intents and purposes, a “Beazley Machine,” that is, a supervised machine learning system designed to mimic and/or agree with any and all of Beazley’s attributions.

Ryan’s data-hungry algorithms function, almost exclusively, by winkling out patterns from examples, and the more examples, the better. For this reason, Beazley seems a very sensible choice for this research. The amount of data and documentation remaining from Beazley’s attributions is impressive by the standards of the humanities. Beazley used in his practice, just as Morelli had, hand-drawn, schematic renderings of the details/features that he deemed emblematic of each Attic vase-painter’s style, drawings that he published in his articles and books (see Figure 4).

Over the course of half a century Beazley came to identify hundreds of vase-painters based on detail-features like knees, hands, and costume. The existence of the large photo archive that Beazley amassed of these ceramics, plus the five published, monumental, volumes of Beazley’s attributions, as well as the archival record of Beazley’s dimensionally-reduced line drawings, convinced Ryan that there might be sufficient data to computationally model Beazley’s method [Beazley 1942] [Beazley 1956] [Beazley 1971].[8] Nevertheless, while the collection of data Beazley created on Attic vases is truly excellent and extensive by art historical standards, Ryan found that they proved insufficient to support the types of machine learning approaches that he had hoped to implement.

The method that Ryan used to solve this problem of data scarcity interests us greatly for what it reveals about the necessary role of human beings in the process of art attribution even implemented computationally. Ryan elected to address it by producing “artificial data sets” that increase the amount of data available for training. That is, Ryan himself sketched a number of prototypical mouths, knees, and ears in the manner of a number of Antique painters, digitized them, and included his own line drawings as part of the Beazleian training data. This technique served very specific computer-scientific ends for Ryan, namely to help discover if training the classification engine on a combination of both original and artificial data reduces the risk of overfitting to a small design set significantly enough to justify the risk of adding non-essential information [Ryan 2009, 100ff]. But, critically, he went so far as to argue that the inclusion of his own drawings — insofar as they gesturally re-enact the work of a trained, human connoisseur — made his computational experiment more accurate.[9] This is, interestingly, in line with Beazley’s own practice. Drawing inspiration from Morelli’s method, Beazley produced schematic line drawings of the forms and figures he observed on Greek vases in the belief that the identity of the artist responsible for painting a given vase might become clear when comparing these schematic surrogates against one another. But occasionally, Beazley begged for the reader’s indulgence as he produced line drawings that were more reflective of his own hand than that of the Greek artist he was attempting to ventriloquize [Beazley 1922, 97]. By inserting his own humanity into his datasets, we find that Ryan is not only following Beazley’s example, but also asserting that trained human actions and decisions play an unassailable role in the process of art attribution. It is Ryan’s judgment that makes this all work, not Beazley’s and not the computer’s.

And yet, when Ryan offers a number of justifications for his work, we feel that most of them are more consonant with the traditions of computer science than the humanities. Crucially, he argues that one of the most fundamentally positive contributions of such a system would be to make the practice of art attribution more efficient, that is, to save human time. Rather than focusing on the central importance of human expertise, Ryan suggests that his system could save connoisseurs from wasting precious time performing “painstaking, difficult” operations, and could free them up to focus on any larger issues at hand [Ryan 2009, 36]. This move to assert that the value of computation in any field is to save time and increase efficiency may function well in domains such as engineering or business, but we find it largely unconvincing in the context of the humanities.[10]

For Ryan, the opposite of an efficient system is a slow, painstaking one. However, in the context of the humanities, this analogy simply does not hold. For a humanist, slow is not necessarily bad and it is certainly not the opposite of “efficient.”[11] The value of the humanities does not directly reside in how quickly it can be done, and indeed the work can be considered un-finishable: full of difficult, fundamentally unanswerable questions that each human generation reassesses anew. We believe that the value of the intellectual work of the humanities partially resides in the fact that it takes time both for human beings to gain trust in one another and also to be trained to make scholarly judgments. Will it, or even can it, “save connoisseurial time” to delegate the practice of, say, visual comparison, to a computer program? Ryan brings no evidence to bear to demonstrate that there are other ways to produce this kind of domain-specific knowledge other than the very practice he is automating, that is, having a human being perform these “painstaking, difficult” tasks, especially those that require “very narrow expertise,” as he himself notes [Ryan 2009, 36].

We do not believe that such a demonstration is possible, and it would be irresponsible to find out simply by trying. The question of whether or not a certain painter painted a given part of a picture is, ultimately, unverifiable. While there is a reality to it, we cannot replicate it as an experiment: we do not have a time machine. What an act of hubris it would be to think that we could at any one point solve the problem of art attribution once and for all! Disagreement and a lack of complete consensus in an interpretive community is not a sign of its failure, or lack of “efficiency;” they are signs of its correct functioning as this community goes about its work of continually producing meaningful, cogent knowledge.

In our digital work with the Morellian Method, we focused not on issues of efficiency, novelty, or even the extent to which we could fully mechanize art attribution, but instead on using the computer to help slow our thoughts down, pick apart what we were seeing, show us some of our assumptions in pain-staking detail, and to dive deeper into the system we were formalizing. Rather than hoping that the computer system would partially mechanize our trained judgement, we hoped the process would allow us to use it more judiciously. We wished to engage with what it would mean to implement the Morellian method within the context of digital computing while maintaining our focus on the intellectual priorities of the humanities, hoping to take advantage of the “computational mirror” to come to learn things about Morelli’s methods and our own. We therefore proposed a technical collaboration with the Extreme Science and Engineering Discovery Environment (XSEDE) group, an NSF-funded, computational and collaborative infrastructure built on top of the United States’ national network of supercomputers, for the development of a proof-of-concept “Morelli Machine” that would take advantage of the team’s interdisciplinary composition to use digital techniques to apply Morelli's method to a large set of images of artworks.

We were keenly aware that the reward structure that would surround the technologist assigned to us by the XSEDE infrastructure would differ greatly from our own as academics, and we wanted to design the details of the project so that it could accommodate both our desire to create recognizable humanistic research as well as fit the requirements of grant-funded technologists who would need to show computational innovation and novel results.[12] Therefore, after our initial proposal was accepted, we designed a two-phase project plan with the input of the XSEDE technologist-consultants Paul Rodriguez and Alan Craig. First, we would apply Morelli’s method in a somewhat strict historical manner. We would ask the computer to extract the eyes, mouths and hands from a dataset of images and then cluster them into formal categories, that is, we would ask the machine to follow the same workflow detailed in Morelli’s writings.[13] We would then look to see what groupings the computer might find, in part to compare the results to the known clusters of artistic similarity, but also to see if the computer was able to group these features into any other sorts of meaningful groups that may not (yet) hold any resemblance to art-historical opinion. We chose to apply a standard computer vision technique known as Histogram of Oriented Gradients for Phase One, rather than either a supervised or unsupervised machine learning system, for two reasons: first, as we will note below, Phase Two was earmarked for such approaches, and second, Histogram of Oriented Gradients, as a feature descriptor that operates similarly to edge detectors, works by isolating dramatic shifts within digital images, an approach that seemed to map well to the process that Morelli proposed.

It was recognized from the very beginning that this work, like Ryan’s, would require data and a great deal of it. We had initially hoped that we could amass between two and five thousand images drawn from a handful institutions, such as the Kress Foundation, The Metropolitan Museum of Art, and the National Gallery. Our plan was to gather this relatively large number of images, or perhaps even more, from ArtSTOR’s collection of digitized and digital images of artworks. While our initial conversations with ArtSTOR seemed promising, in the end, the red tape surrounding the large-scale use of multi-institutional images proved insurmountable to solve in the time we had to address it.

However, Nygren and Langmead knew, because of prior digital humanities work their students Andrea Maxwell and Sarah Reiff Conell had done with the images of the Samuel H. Kress Collection, that the Kress Foundation is very generous in sharing their collection of digital images for computational use, and moreover, offered dependable metadata. After our initial conversations with ArtSTOR faltered, we approached the leadership of the Kress Foundation with a request to use their images for the purposes of this project, and they happily agreed, while noting that perhaps the easiest way to acquire the files would be, ironically, through ArtSTOR. With the help of the Foundation, ArtSTOR was able to provide us with the digitized images of the Kress Collection, a dataset that amounted to 1,866 files, not “Big Data” by computer-scientific standards, but too large and at too-high a resolution for a laptop to process, and was sufficient to kickstart our work.

For the second phase of the project — one we will not have space to fully detail in this essay — we recognized that Morelli’s approach of decomposing the images into distinct human-recognizable features would not be the way that contemporary technologists might choose to approach this work if the goal was clustering and classifying images by artist. It is currently quite difficult and time-consuming to ask a computer to do the work of extracting human-defined, visual details from images, and it only has marginal computer-scientific novelty to approach the problem in this manner. And so, Langmead, Nygren, Rodriguez and Craig collectively designed a plan for Phase Two that would compare and cluster the images in this same dataset using machine learning approaches that let the computer select the important features, rather than forcing the use of human-defined ones.[14] Even though we will not detail the process or the results of this second phase in this paper, we mention it here because we feel that it is important to present the entire scope of the collaboration and the balance we tried to strike between serving the working needs of the humanists and the technologists on this project. Our results from Phase Two will be the focus of a separate study at a later date, and has already been presented by Rodriguez to the XSEDE community and received with interest.[15]

Phase One of the “Morelli Machine” project began in August 2018 and concluded in January 2019. Given that the off-the-shelf digital technologies for facial recognition were, and are, far more advanced than any other type of human body-part-detection, we focused on the process of extracting facial features such as eyes and mouths from the data set, rather than hands or feet. Step one on this path was, therefore, to extract faces from the images. Rodriguez began by trying the Dlib Face Detector and OpenCV Face Recognition libraries, but found in the end that the Google Vision API was the most successful at identifying the faces within our dataset.[16] Google’s technology was then applied to the entire collection, and succeeded in extracting 3,205 faces. This API also returned facial landmarks — that is, polygons that outlined its best guesses as to where the eyes, nose and mouth were located on each face it identified — and Rodriguez used this information to post-process the results and extract our desired facial features as individual, cropped images (see Figure 5).

To fine-tune this process of feature extraction, Rodriguez applied pose (i.e., head turn) estimates, again using the Google Vision API, to adjust the size and shape of these boundary polygons. We wanted to be as careful as possible to extract only our desired facial features, knowing that applying Morelli’s method requires accurate feature extraction. Overall, to the human eye, the results appeared quite good. Rodriguez estimated that around 71% of the faces found within our image files were successfully identified, and within that result set, we were able to extract all of the eyes and mouths [Rodriguez 2020b].

Sufficiently satisfied with the accuracy of the feature extraction, the team then moved to the task of clustering. The team began by focusing on the mouths and Rodriguez turned to the Histogram of Oriented Gradients technique to begin work. Histogram of Oriented Gradients (HOG) is a feature descriptor that operates similarly to edge detectors, that is, rather than treating images as color values, it works by identifying the direction and magnitudes of relative difference between pixels. These differences can then, in turn, be used to detect the edges of and/or identify the human-recognizable objects represented in the images.[17] At each pixel, the algorithm calculates a vector for the “gradient,” that is the amount of numeric change between that pixel and the other pixels adjacent to it horizontally and vertically. Those vectors are then added together to create a main direction and magnitude for each pixel, that is, its oriented gradient (see Figure 6).

When implementing HOG, the meaning of “change in pixel value” can be tweaked by the technologist to best identify those features of an image that may be more important to the study at hand. This could be color detection, saturation detection, or any other fundamental, numerically represented, feature of a digital image. To work effectively, therefore, a bit of data pre-processing on the images is required to highlight those characteristics that it has been decided are important to the assessment at hand, and to deemphasize those felt to be extraneous. Deciding what was and was not visually important to implementing the Morellian method computationally became something that our team needed to identify at the outset. As art historians focused on implementing Morelli’s method as closely as we could, Nygren and Langmead were very concerned with maintaining as much color information in our analysis. Rodriguez took this request into deep consideration and opted to normalize the eye-images so that the total illumination was constant but relative color values were still present.[18]

As a next step in the HOG process, all pixels in the image are clustered into blocks of four and the vectorized gradients for each of them are then used to calculate nine summary gradients as measured from the block center (see Figure 7). These nine summary vectors are then counted and visualized as a histogram showing the distribution of the sum of their magnitudes, categorized by their orientation. The greater the count in any given bin, the “stronger” the gradient is said to be.

The penultimate result of the Histogram of Oriented Gradients feature descriptor, then, is that the images have been decomposed into a set of clustered, oriented, summarizing, nine-dimensional vectors that describe the relative numeric change in pixel values across the image, one that also retains information about the relative direction in which that change occurs (see Figure 8).

These vectors can then be visualized as a histogram that can be taken to assert the relative “strength” and “weakness” of the gradients across the image. To conclude the process of applying the Histogram of Oriented Gradients technique, a summarization method is applied: one final feature vector is calculated by taking all of these nine-dimensional vectors and summarizing them into a single, large, ten-dimensional vector that represents, in some ways, the image in its entirety [Rodriguez 2020a].

Once this process had been run on our extracted features, it was time to arrange them into clusters of similarity. The ten-dimensional HOG feature vectors that summarized each image were used as input into a K-means clustering algorithm to gather the images into (hopefully meaningful) groupings. To visualize this information, the original ten-dimensional feature vectors were projected into two-dimensional space using the t-SNE technique, and each point on the resulting graph was then color-coded according to their K-means clustering assignments (Figure 9).[19] We began with mouths.

The initial clustering results for mouths did not inspire anyone on the team to assert that our computational method was successfully grouping the images by any clear, meaningful art-historical metric whatsoever (see Figure 10). Above and beyond the fact that the team had clearly produced an excellent Old-Master-Painting mouth extractor, the clustering results produced by the combination of the t-SNE and K-means procedures was not anywhere near to clustering these features by artist. None of us were even particularly struck by any amount of above-average visual similarity within the clusters. The results for eyes were no better.

It was becoming clear that HOG was not sorting these images into groups that held visual interest for the team. But even so, because our own research questions asked us to make some form of art-historical, or even visual, sense of the clusters of disembodied mouths, Langmead and Nygren were afforded the time and the opportunity to acknowledge and respond to their own immediate and visceral recognition of the aesthetic violence of Morelli’s method, one perhaps magnified by the scale offered by the digital computer. This moment offered us an experiential taste of what it means to extract recognizably visual features from art objects and array them like so many specimens in a row. Moreover, it suggested how hard it can be to visually interpret these isolated images at this scale, one orders of magnitude more massive than Morelli could have possibly imagined. Morelli himself suggested that it was overwhelming to confront dozens of sketches of fragmented body parts, and seeing thousands of them proved no less disconcerting (see Figure 11).

When comparing Morelli’s published line drawings of disembodied hands and ears to our data, Langmead and Nygren had been considering these visually comprehensible images as the computational analogs to Morelli’s Grundformen. However, during a team meeting, as Rodriguez took a moment to narrate his process in greater detail, he also shared the set of pre-processed images created for the HOG feature descriptor. It was at that moment, with the grids of mouths in mind, that Nygren and Langmead were suddenly struck with the realization that these pre-processed images — not the disembodied, recognizable mouths that had been extracted by the Google Vision API — were the objects analogous to the Grundformen, the dimensionally-reduced line drawings used by Morelli (see Figure 12). These (to human eyes) yellow and white, almost wholly abstracted forms, were the more fundamental visual evidence of the extreme computational abstraction necessary to make claims about sameness and difference within a computer, not the grid of human-identifiable mouths.

To explain what we mean let us return to the Beccadelli portrait we introduced above, because it is in the illustration accompanying his discussion of that painting that the ultimate stakes of the Grundformen become clear. Through working with Morelli’s method within the context of digital computing, we came to understand that it is predicated on a misdirection that leads us away from the painting-in-the-world toward an abstract, schematic rendering conjured into existence by the connoisseur. Closely comparing Morelli’s line drawing to the Beccadelli portrait, we noticed that his sketchy rendering is quite deceptive and does not appear to duplicate either hand found in Titian’s painting. To a trained art historian, the hand seen in Morelli’s line drawing actually seems much closer to the hands found in numerous other TItian paintings executed around this same period in Titian’s career (1540-50), such as the Crowning of Thorns, The Allocution of Alfonso d’Avalos, or the Portrait of Vincenzo Cappello (see Figure 13). For a man focused on details, Morelli appears entirely uninterested in offering direct visual parallels between his line drawings and the particulars of this portrait [Uglow 2014] [Brewer 2018].

Elsewhere in his volume, Morelli suggested that the Grundformen should resolve into a “physiological treatise” [Morelli 1893, 10]. Having trained as a physician, Morelli’s Grundformen “identify” the author of a painting in the same way that a diagnostic manual “identifies” a disease: if diagnosis relies on inference based on symptoms, so too does attribution mobilize evidence in the service of inferential reasoning that is ultimately metonymic.[20] It is important to understand that despite Morelli’s insistence that the Grundformen constitute a form of ground truth on top of which he constructed his attributions, his practice made apparent that they are significant personal abstractions intended for use by trained professionals who have a great deal of pre-existing visual expertise. The Grundformen do not exist in any real sense, that is, the Grundform of a Titian hand is not the same thing as a photographic detail of Beccadelli’s hand as painted by Titian. Rather, the Grundformen were, in Morelli’s account, a wholly schematic rendering of the type of hands formulaically produced across Titian’s entire pictorial corpus, which existed firstly in the mind of the scholar and only secondarily as line drawings.

This realization forced us to consider Morelli’s texts in a new light. On the basis of what we learned by attempting to computationally recreate Morelli’s method, it was no longer clear to us that we were making correct assumptions about how he expected his books to be read and what role he intended the Grundformen to play in the process of attribution. On the one hand, Morelli could have expected his readers to take copies of his book to the unattributed painting and compare his line drawings to the image in vivo. On the other, his books could have been taken as an invitation to the reader to produce abstracted line drawings of their own, only comparable to Morelli’s published drawings, to use in the service of their own attributions.

At this point we feel it is worthwhile to underline what we consider one of the major findings of this project, at the level of both art-historical and computational analysis: Morelli’s Grundformen are not simply dimensional reductions of visual data. The Grundformen are embodied renderings of human judgment, the result of hundreds (if not thousands) of hours of deliberatively looking at early modern paintings with the larger intention of developing a method for describing artistic style. Simply looking at an array of extracted images would never result in the production of a Grundform; to do this work, a human must exercise trained judgment to identify precisely what is and what is not important to their own informed understanding of the notion of artistic style and then use those concepts to extract only what is necessary to them. There is no promise that the Grundformen of one scholar would match those of another. Similarly, asking a computer to replicate this workflow cannot be achieved by training the computer to recognize, say, every human hand painted by Titian; rather, it would require the computer to produce its own abstracted rendering of Titian’s “hand,” which encompasses both the hands he painted and his stylistic “hand,” and then to deploy successfully that abstraction identifying artistic style. While current computer techniques are becoming capable of achieving the first step in this process, as represented by the yellow-and-white abstractions made for the Histogram of Oriented Gradients workflow, the higher order function of articulating how these abstractions make meaning in the context of the scholarly community remains beyond its reach without human intervention. For the machines to be able to attribute artwork on their own, we would need to allow them to join our intentional community and delegate some of the responsibility for our judgment to their mechanical processes.

We set about this collaboration because we were curious to see what it would require to implement an art historical method of connoisseurship computationally. We have determined that, for the moment at least, computational techniques cannot attribute artworks without the support of human judgement and expertise.[21] While this situation may change as our relationship to technology transforms over time, our core belief remains that looking to a computer for the “answer” to the question “who painted this picture?” is currently not only misplaced but can also have noxious implications.

Although we did not initiate this collaboration with the art market in mind, it quickly became clear to us that our project would not be perceived as agnostic by that market itself and the stakes of this work partake of this larger ecosystem [Elgammal et al. 2018].[22] This perhaps should not have come as a surprise, since Morelli’s method has had a marked impact on the art market [Agosti et al. 1993]. Because the price of an object on the market is interlinked with other concepts like historic import and artistic “value” (however defined), any mode of verifying the authenticity or attribution of a work of art that is deemed to be trustworthy will, almost as if through capillary motion, flow into the surplus value of the work of art. Over the course of the twentieth century, increasing preference has been given to the evidence produced through technological processes to determine authorship [Carrier 2003] [Wollheim 1973] [Grabar 1988]. In 2016, Sotheby’s purchased the firm Orion Analytical, making them the first auction house to create an in-house scientific lab dedicated to authenticating works of art [Subramanian 2018].[23] The auction houses have sought refuge in the seemingly dispassionate results of scientists, a form of evidence that takes on the air of objectivity and thereby is granted authority, especially by those outside of the scholarly community.

Carrying this tradition forward, computer science researchers have recently been promoting their technical skills with computer vision to hold out the promise of “art verified by AI.”[24] Such interventions draw on the same desire for “objective” data as earlier attempts to incorporate scientific results of pigment analysis and carbon-14 dating, but this time wrapped in the rhetoric of artificial intelligence. The assertion that an attribution has been “computer verified” is likely to increase the value and prestige of old master paintings. Would collectors be more likely to wager the GDP of a small nation, as in the example of the Salvator Mundi above, on the conflicting opinions of embodied art historians or on the declarative output of a computer algorithm? We cannot claim to know with certainty, but it seems as likely as not that a buyer will trust the definitive answer produced by the black box of a computer algorithm over a cacophonous exchange of interpretations between the competing voices of art historians. The art market demands answers, not a series of questions.[25]

As humanists, however, we believe that such questions are truly important. What does the insertion of a computer change about, or add to, the traditional humanistic process of looking at objects and providing well-founded, inferential statements about the past? Is it the immense speed of calculation? Is it the size of the data sets? Could it be that people might trust a computer in a way that they no longer trust intellectual forebears such as Morelli? We have engaged in academic research, and to the extent possible we have attempted to sever our inquiry from the “umbilical cord of gold” that links art, art history, and capitalism.[26] However, our community’s efforts in this domain must remain ongoing.

Research in this field should not only resist instrumentalization by the art market, but also offer perspectives on the serious stakes of what it would mean for human beings to vest computers with the sole responsibility for making attributions. Let us return once again to the thought experiment proposed in the introduction in order to make a specific point. To whom would the purchaser of the work purportedly by Leonardo make recourse in the event that the algorithm that produced a computational attribution of the painting to the master was proven to be faulty? The culpable might include: the computer programmer who developed the algorithm; the art historians (if any) who contributed to the construction of the algorithm; the lines of code in the algorithm; the piece of hardware that ran the algorithm; the auction house that countenanced the results; the buyer who believed the results; or the insurance agency that indemnified the results. In this day and age, it is not possible to imagine that, from such a list, anything other than a human being can be held accountable for such an error, and yet the decision would have been “made” by a machine.[27]

The difficulties faced in attempting to respond to this scenario point, we believe, to the unavoidable truth that computers have not earned their place in the conversation about artistic attribution because the linkage between “style” and authorship is not some immutable law of physics that is easily computable, but rather a humanistic commitment that requires human judgment. To make the practice of art attribution anything else besides a necessary collaboration between the physical object and trained human beings would require a fundamental reassessment of the practice, not simply its mechanization. We would like to argue that all such systems will currently fail to produce convincing results, not because they are incapable of mimicking the formal methods of art attribution, but because they have not (yet) been accepted as members of the community that make up attribution’s social practice, and cannot take responsibility for their conclusions. The computer can present the art historian with a series of statistical probabilities related to authorship, but it is important to note that simply choosing between a menu of options presented by the computer is not sufficient for producing a genuine attribution: art attribution cannot be reduced to decisionism; it requires judgment. We affirm that it is necessary for the community of experts called upon to make these judgments be made up of a broad and representative population of human experience that goes beyond the demographics that have traditionally dominated the space of attribution. It is precisely because human judgment is necessary to the process of attribution that, to make effective meaning, the community must reflect the breadth of the human experience.

While the results of our Phase One workflow were not designed to “give answers” to the problem of attribution, and were certainly not successfully responding to that imperative in any way, they have illuminated the ways that the mechanical processes of abstraction that, at first, seemed similar between our computational approaches and Morelli’s, were in fact utterly different, once contextualized within a social system. When they are presented as abstract entities on the page, Morelli’s Grundformen appear reified; through rhetorical force, they seem to take on an existence that was independent of the paintings in which they may be found. This presents an occasion for “verification” and cross-reference that can be described as quasi-computational and therefore formalizable. But the process of art attribution does not begin and end with the physical materiality of the painting.

Attributing a work of art has always been the work of narrative explication rather than mere reckoning about stylistic markers. It currently requires human judgment, expertise, and trust to become an effective truth in the world. Morelli, too, gestured towards this state of affairs by presenting his method in his text as a product of dialogue, as a conversation embedded in expertise and interpersonal persuasion. Morelli’s Grundformen existed to help his interlocutors see the paintings as he saw them, as a way to convince others of his judgments. Even granting that computational methods eventually rise to the level of being able to make believable assessments of stylistic markers, for their outputs to become truly effective, they would need to earn their place as full participants in the social process of art attribution, and human beings would need to delegate some power of judgment to them.

When we began this project, we took a leap of faith that we would uncover interesting areas of art-historical inquiry by computationally mechanizing Giovanni Morelli’s method of art attribution. While it was to be expected that our system could match Morelli’s own at the level of the workflow — extract a set of details and compare them — we did not anticipate how our work would show us the complexity of Morelli’s historical process at a more fundamental level. Where Morelli used his native human powers of abstraction and judgement to reduce old master paintings to a series of published line drawings, we found ourselves using the computer’s power of abstraction to create seemingly analogous, dimensionally-reduced representations. In so doing, we recognized that Morelli’s abstractions were effective because they worked in parallel with his human judgment in a way that mechanical processes could never replicate alone.

Our investigation has also suggested to us that there is a fundamental question that the field of art history has not yet addressed and that computers might actually help clarify: to what extent are personalized, identifiable artistic traces, however they are defined (forms, brushstroke, pigment, craquelure, etc...), registered at the level of pixels? Because, insofar as these traces are so registered, they will be legible computationally and therefore computers can identify them, perhaps even more effectively than human beings. We nevertheless argue that pixel-level information, on its own, would be insufficient to produce a credible attribution. That work is done as a social practice; it is a matter of an interaction between the physical traces of the past on the canvas and the trained judgment of human beings. It is an ecology that includes the physical history of the work of art, the archive of historical knowledge external to the work itself, and the particular training of each expert working in a community that has the power to make their findings real and effective in the world.

Classical, deterministic, electronic, digital computers are designed to produce certain results based on the input given and their own physical and logical programming. They only show you what you have set them up to show you — not necessarily what you wanted, needed, or even expected to see. Humanists natively operate in a different space than this, which might suggest that the world of computation cannot speak to the world of the interpretive expert. However, we argue here that this is not the case. Working with the computer, and with trained technologists, has provided us with the opportunity to gain important perspectives by allowing us to produce and manipulate an art-historical model of the world. A truism that has become particularly self-evident during the COVID-19 pandemic is that all computer models are, in some senses, wrong, but some models are useful.[28] Recognizing the ways that our computational model did and did not implement the older art-historical approach was, for us, like both looking through a microscope and a mirror onto the history of our discipline. This exercise forced us to slow down and look at Morelli’s method in minute detail, but in the end, it also reflected our own assumptions back to us, providing us with clear critical distance. Using digital techniques to think through even historically distant modes of art history indicates one path by which our field can do the ongoing work of reshaping its own practices from within while also reaching out and collaborating with other expert communities that see the world differently. Bridging those gaps, we believe, is crucial to developing art history for the digital age.