Isabella d’Este: Virtual Studiolo borrows its conceptual framework from four domains: the humanistic field of Renaissance art history; the theory and practice of museum curation; current thinking in cultural heritage reconstruction about “sentiment analysis,” and the technical-creative disciplines of digital modeling, animation, remixing, and remastering. The project aims to reassemble in a virtual, open-access environment one of Renaissance Italy’s most stunning courtly art collections, the studiolo of Isabella d’Este (1474-1539), marchesa of the city-state of Mantua. The Virtual Studiolo will join a set of already existing, interlocking projects gathered in the multimedia online environment of IDEA: Isabella d’Este Archive (http://isabelladeste.web.unc.edu) (fig. 1). As a complex, IDEA provides resources for the study of the Italian Renaissance through the figure of one of its most distinguished and celebrated women, Isabella d’Este.

Breaking with much more modest practices by noblewomen who owned small galleries of family portraits and religious images, Isabella set out shortly after her marriage (1490) to Marchese Francesco II Gonzaga to design a full-blown personal studiolo (study) and an adjoining smaller room she called the grotta (grotto). These camerini (little rooms) functioned as personal retreats for reading and reflection but also as galleries for the display of Renaissance learning and luxuries and as private performance spaces for Isabella’s music. A purposeful projection of her ideal self, they were created both for her own pleasure and for exhibition to privileged visitors [Liebenwein 2005] [San Juan 1991] [Cartwright 1903]. Isabella managed her acquisitions largely through voluminous written correspondence with artists and dealers, directing the lavish decoration of these rooms with intarsia panels, gilded ceilings, frescoes, tiles, and enigmatic personal emblems. She also filled them with treasures she collected, including books, cameos, Roman antiquities and Renaissance bronzes. Dominating the studiolo was a series of large paintings that now hang prominently in the Italian galleries of the Louvre: two each by Andrea Mantegna, Lorenzo Costa and Antonio Allegri da Correggio, and one by Pietro Perugino.

As a signature instance of personal “branding,” Isabella’s studiolo stands on a historic par with Granduke Francesco de’ Medici’s later studiolo in the Palazzo Vecchio in Florence and those of Duke Federico da Montefeltro in Urbino and Gubbio (the latter now transplanted to New York’s Metropolitan Museum of Art). It also holds special significance as a collection that anticipated the achievements of such modern female collectors as Isabella Stewart Gardner in Boston, Etta and Claribel Cone in Baltimore, and Peggy Guggenheim in Venice. Isabella’s camerini were an unparalleled architectural and artistic expression of specifically feminine Renaissance culture. In their sensory and intellectual density, they functioned as a space for contemplative withdrawal but also for display, and as a controlled environment in which the human passions could be summoned and engaged while also remaining safely contained [Campbell 2006]. One of this project’s aims is to allow visitors to experience the sensory power of the studiolo in a virtual environment.

Both for its coherence and for the gender of its creator, Isabella’s studiolo is a regular point of reference in the study of Renaissance history and art, yet for centuries it has been accessible only in dispersed pieces and spaces depopulated of major works and artefacts. As documented in detail by the late Clifford M. Brown, the rooms remain but have sustained significant renovations; many of the studiolo’s paintings and sculptures survive, but they belong to museums around the world [Brown 2005].

Given the dispersal of their Renaissance contents, it has been impossible to approximate the experience of these rooms as fully appointed and purposefully designed. Today researchers, students, and visitors are limited to looking at books, slides, and variously published online images; they may also listen to limited examples of recorded music. Visiting Mantua’s Ducal Palace in person, on the other hand, one finds empty rooms whose former glory can only be imagined. For a brief tour of the rooms of Isabella’s second studiolo as it appears today, see the video “Ad tempo taci’: Songs for Isabella d’Este” (see 6:00 ff.) [MacNeil 2015].

Current digital technology — photogrammetry, 3D modeling, the virtual reassembly of dispersed collections, and augmented reality display — offers the possibility of creating a “remastered” studiolo, a virtual space in which both visual and acoustic elements will be enhanced with respect to previous attempts at its representation. Yet historical uncertainty about numerous details in Isabella’s arrangement of the objects in her collection compels us to present, and to argue, this project as itself a hypothetical and interactive remix: a media artefact that inevitably alters the original through processes of reconstruction, supplementation, subtraction, and reorientation to create something unquestionably new.

Adding to the complexity of efforts to visualize the historic studiolo is the fact that Isabella had it constructed twice, in two different locations at different times in her life, and the two versions of these spaces are in different states of repair and restoration. If one premise of our endeavour is that the virtual integration of the studiolo’s contents will foster understanding of collecting and display, of the studiolo as an early modern project, and of Renaissance culture broadly conceived in ways that have been unattainable for centuries, another is that a single, definitive, and statically “authentic” restoration of these historic spaces is not only impossible, but also undesirable. We view the Virtual Studiolo as an opportunity to acknowledge the dynamism of collecting and curating as cultural practices, today as well as in the sixteenth century.

Accordingly, we are aiming to present both our own curation of the studiolo as it has been researched and documented by art historians, and also the opportunity for users to move objects in and out of the spaces, rearranging and adding contents in newly-authored configurations — remixes — that may be saved for personal use or publication. Currently we anticipate the use of Blend4Web (https://www.blend4web.com) as the framework both for navigating our own, proposed arrangement of the historic studiolo and for programming a customisable environment for future users. In anticipation of this phase of the project, the Samuel H. Kress Foundation (www.kressfoundation.org/) funded a concept-demonstration video that allows us to present some of the project’s foreseeable further developments. We would welcome feedback on this material, which users may offer either by contacting us through the IDEA project’s home page: (http://ideaart.web.unc.edu/virtual-studiolo-studiolo-virtuale/), or by filling out our short User Story survey: https://docs.google.com/forms/d/1nbhkt7W0nHwVh76EOiQNtH-IfkKlRARPtD7UAoKzPVM/viewform?edit_requested=true .

The design and development phase of the demonstration video for Isabella d'Este: Virtual Studiolo aims to illustrate concisely and effectively to potential funding agencies the attractions of a complex project such as the Virtual Studiolo, while also raising awareness regarding the range of disciplinary collaborations and knowledge bases required for its realization (http://ideaart.web.unc.edu/virtual-studiolo-studiolo-virtuale/). For this purpose, we chose to produce an emotionally captivating video presenting both the historic space of the studiolo, as a 3D reconstruction resulting from a photogrammetric campaign, and a small selection of objects from the collection once housed in it. The video, for example, opens on a vortex of Isabella’s letters, which alludes to the surviving correspondence that documents the studiolo’s acquisitions and that is viewable in another IDEA project, IDEA Letter/e. It shows two musical instruments represented by intarsia panel carvings in the historic studiolo — a lute and a spinet — which recall the love for music that led Isabella to commission songs that she played on those instruments (and which are the focus of several projects in IDEA Music/a). It shows ceramics that she received as a gift from her daughter (featured in the IDEA Ceramics project located in IDEA Art/e). And it also features a high-resolution rendering of one object from her collection, a precious, jewel-encrusted medallion engraved with her portrait, for which image we benefited from the engaged collaboration of Vienna’s Kunsthistorisches Museum (Figure 2), and lower-resolution renderings of several sculptures belonging to her collection. The studiolo is thus presented as a full-scale Wunderkammer devised by a woman who has been called the “first lady of the Renaissance”  [Bini 2001] [Ferino-Pagden 1994] [Pizzagalli 2001].

Our challenge was to present in just under six minutes the state of the historic studiolo today and, at the same time, to simulate projected phases of implementation that will involve virtual reconstructions, assembly of dispersed objects, online exhibitions, and several other tools that digital technology can offer for preserving and accessing cultural heritage, in particular for educational purposes (Figure 3). We sought, as well, to present the range of research options that a remastered studiolo will offer, and to integrate our protagonist, Isabella d’Este, whose portrait at a certain point peers out from the video as if to claim her role as “author” of the studiolo, as a driving force behind its decades-long ideation and construction.

Conceived as an introduction and invitation to the project, the video is a compact whole that is separated into two parts. The first, more creative, emotional and evocative segment is accompanied by an insistent, contemporary music track by Kai Engel that carries a certain driving force. The second segment includes a sound track of music from Isabella’s time. This sequence is especially important, as it conveys both the participant credits, which range across disciplines and institutions, and technical information about the potential evolution of the project. The video’s conclusion opens out onto new horizons for study and educational itineraries that are now possible thanks to digital media. Our project is therefore consciously a marriage of the current renaissance of art and technology with the historic, Italian Renaissance from which we draw inspiration and knowledge.

The creation of the video began with a photographic campaign for the photogrammetric acquisition of the historic studiolo and grotta in Mantua’s Ducal Palace. More than 500 photos were taken inside each of Isabella’s two rooms at a 36-Megapixel resolution. The photogrammetric reconstruction, with PhotoScan (www.agisoft.com/) (Figure 4), produced the following results:

• for the studiolo, starting from 574 photos, the software was able to align 574, creating a dense point cloud of 110,945,015 points with 11 million vertices and eight 4096*4096 pixel textures;

• for the grotta, starting from 620 photos, the software was able to align 492, creating a dense point cloud of 122,408,100 points with 12 million vertices and four 4096*4096 pixel textures.

The 3D models were then imported and optimised in Blender. The mesh was duplicated and divided into macro areas (ceiling, walls, floor) and further subdivided into smaller zones. On the bare walls we applied a planar decimation, while on the rougher or more irregular areas of the mesh — such as those corresponding with the intarsia on the ceiling — we applied a collapse decimation. The output of this process was a mesh with a lower level of detail (100/200.000 vertices) that is suitable for full-room renderings. For close-up shots, instead, segments of the full-resolution mesh were used.

As for the textures, since PhotoScan does not optimize UV unwrap, the mesh had to be manually subdivided through seams, in order to isolate planar areas and perform a new manual unwrap. Two techniques were applied. Blender standard unwrap was used for the more irregular portions, with a relax post-processing passage for better smoothing out of irregularities. The second technique we adopted was a project from view.

In some areas, such as those with the intarsia wooden panels or the richly coffered ceiling, we obtained a higher level of resolution by scaling the UV island; for an immediate visual feedback we applied a 4K resolution UV grid texture (Figure 5). The mesh material was configured in order to use a new empty texture for the colour. In this instance, the colour information obtained with bake was transferred from a high poly mesh to an optimized one.

Several other techniques were also key for the video: fluid simulation, for ink and sealing wax drops (Figure 6); a series of particle simulations, for the vortex of letters; Blender animation nodes for the spreading of the floor tiles; and 3D photo matching, for Isabella’s portrait.

For a credible representation of the ink and sealing wax drops, the Blender fluid simulator was programmed with the real physical density of the two substances. Moreover, for sealing the wax, Blender modifiers and shapekeys were used, as explained in the tutorial we have shared on the VisitLab blog (http://visitlab.cineca.it/?p=1771 ).

As for the vortex of Isabella’s letters, three main particle systems and two auxiliary ones were created, the latter for adding random dynamics. The main particle systems created the three movements of the shot: the ascending movement of the letters, the vortex and its explosion (Figure 7). A further particle system, with keyed particles, was added to these three for managing the transitions among them. Images of five of Isabella’s actual letters were used instead of point particles.

For the animation of the floor tiles, a significant contribution came from the online community, thanks to the add-on “Animation nodes” (https://github.com/JacquesLucke/animation_nodes) and Archimator, a pre-set of configurable nodes for the creation of groups of objects: (http://blenderartists.org/forum/showthread.php?350296-Addon-Animation-Nodes&p=3013604&viewfull=1#post3013604) (Figure 8). The Titian portrait of Isabella d’Este was recreated with 3D photo matching using the tool Make Human (www.makehuman.org/).

Our desire to create a short video that would convey the many possible applications of Virtual and Augmented Reality technologies that the Virtual Studiolo project could engage, including gaming and 3DWeb and that would, at the same time, entice viewers and involve them in a kaleidoscope of these possibilities, led us to an extensive use of visual effects created in Blender: animation, paint-brush based scenes, morphing, plays of light and shadow and, as noted above, the physical simulation of fluids and the use of node compositing (Figure 10).

All that we have achieved during the creation of this concept demonstration video will constitute a solid basis for future developments of the Virtual Studiolo project, but it also demonstrates more generally the use of environments, objects and animations that is possible with current visualization and interaction technologies, some of which are briefly suggested in the visuals accompanying the closing titles of the video.

Immersive Virtual Reality (VR) technology offers unprecedented tools for meaningful encounters with historic sites and virtual restoration of structures that have been damaged, altered, or destroyed over time [Barcelo, Forte, and Sanders 2000] [Roussou 2002] [Dylla et al. 2010]. Photogrammetric surveys of surviving architectural spaces and objects, digital animation, 3D modeling, virtual acoustics, mapping, and digitized documentation enrich these new forms of access to historically remote artefacts. Yet persisting in these new technologies are enduring questions regarding all historical representation. Whose perspective is reflected in the finished product? How open is it to interpretation and revision? What active roles can users play in the VR experience? How can these products move beyond mere illustration or single-narrative recreational experiences to inspire new knowledge? Isabella d’Este: Virtual Studiolo aims to build a multi-sensory, document-integrated, interactive environment that confronts these issues. It will highlight scholarly understanding of this celebrated studiolo (including uncertainties, which abound), while also inviting user experimentation, interrogation, and customized storytelling to offer a dynamic contemporary experience of the Renaissance culture that the studiolo exemplifies [Liebenwein 2005].

Museums and cultural sites aim to provide meaningful learning experiences [Witcomb 2015] that, as an outcome, very often produce a sort of paternalistic form of “educative leisure”  [Franklin and Papastergiadis 2017] that appeals to a narrow audience. In order to broaden the audience and achieve better results, the communicative approach is gradually shifting toward a different perspective. Understanding that learning and feeling are entangled, that emotions play a role in attention and memory, recent research has increasingly stressed the importance of emotive experiences in learning [Staus and Falk 2017]. We consider emotionally evocative applications to be crucial for the future developments of Isabella d’Este: Virtual Studiolo as well. Our video itself relies upon a visually compelling first segment before giving way to a second segment that presents more direct illustrations of devices, technologies, and products, suggesting our aspiration to mobilize these to communicate elements of Isabella’s Weltanschauung (a new inflection of what art historian Michael Baxandall termed “the period eye”) [Baxandall 1972], by running plausible simulations of the studiolo and grotta environments. Additionally, it shows that 3D virtual environments may be connected to digital archives of letters, music and objects from Isabella’s life and, vice versa, by means of bidirectional paths and interactive, personalised setups.

Numerical simulations, which are an established practice in the sciences, allow us to re-create plausible scenarios that increase our knowledge of natural phenomena by means of mathematical and physical models under different initial conditions and constraints. Similarly, taking as a point of departure IDEA’s digital archive of Isabella’s letters, we illuminate our documentary sources for understanding her early modern world through immersion in a 3D reconstruction of her studiolo, aiming for a scholarly and creative experience that lies in an overlapping dimension between historical research, game and dream. In this project focused on Isabella d’Este’s studiolo, finally, we confirm lessons learned at VisitLab from our previous experiences with the concept of Talking Monuments, where databases and surviving artifacts are merged in a virtual environment [Borgatti et al. 2004], as well as insights stemming from Cineca’s numerical simulations of the eruptions of Vesuvius [Guidazzoli et al. 2006].

We have already tested several types of navigation and user participation employing a variety of technologies, most immediately within the Blend4Web framework. Appropriate adjustments to models and textures for a real-time performance, combined with JavaScript-HTML5-CSS3 programming, will allow users to move around inside the Virtual Studiolo, to interact with objects and writings, to read historical information and metadata, and even to customize the explored environments without needing to install any particular application. Any browser supporting WebGL, even on handheld devices, through a Web address and in a fully transparent way, will provide the user with all the resources necessary to run our application (Figure 9).

Emotionally compelling Virtual Reality experiences have the potential to appeal to a wide and diverse audience and to achieve long-lasting learning impacts. Firsthand testimonials, oral histories and memories are typical means for achieving these effects, in Virtual Museums as well as in traditional exhibits. Where such authentic responses are unavailable, scripted storytelling may provide information that engages visitors, as for example in the Etruscanning project [Pietroni and Adami 2014], where narration is carried out by two fictional versions of the people buried in the Etruscan so-called Regolini-Galassi tomb. Interactivity is a further means for engaging visitors’ undivided attention; adding immersivity to the experience promises to achieve even higher levels of involvement, with interactions that are attainable either in real-life settings or in virtual ones, or in both at the same time [Dong et al. 2017]. As Perry et al. observe, “no coherent framework of practice (neither a conceptual model, nor practical guidelines) yet exists for designing and evaluating emotive experiences for the cultural heritage sector at large”  [Perry et al. 2017]. The Virtual Studiolo will accordingly aim for a range of possible outputs designed to elicit positive user feedback. We will privilege narration, interactivity, immersivity [Champion 2016], and aesthetic/emotive involvement for all visitors to the Virtual Studiolo, while also offering powerful new tools for teaching and research on the Italian Renaissance.

Integration of the studiolo’s surviving features (tiles, doorways, ceiling decoration) with its former contents (paintings, instruments, sculptures, books, clocks) will support research on Renaissance design and collecting. “Moveable” contents will acknowledge scholarly debate about their original arrangement and allow users to test hypotheses about display and curation. At the same time, we also anticipate that users may experience the Virtual Studiolo’s space and objects as a “playable medium,” exploring through a game-like environment the choices, intentions, and tacit judgments that may have informed Isabella’s decisions [Wardrip-Fruin 2005]. Interfaces with other IDEA projects will encourage deeper exploration of the relations between the collection, the collector, and her historic cultural milieu.

For a more immersive experience, both the studiolo and the grotta will be available in HMD applications, with specific exports of the scene designed for navigation with headsets (Figure 11). We also anticipate the creation of special projections inside physical immersive spaces, such as CAVEs, coupled with user tracking to enable a natural-feeling interaction with the objects contained in the Virtual Studiolo.

Finally, Isabella d’Este Virtual Studiolo is conceived as a multi-location, cross-media opportunity that aims both at the general public and at a specific audience of researchers. Our reflections on this spatial multiplicity and its potential realizations include not simply the two levels of real and virtual space, but also multiple potential real interfaces within the current Ducal Palace in Mantua, with its empty, historic studiolo, and any other reproductions of the studiolo that might merge real and virtual layers. In the case of the real studiolo, for example, the museum’s director has considered the possibility of installing physical copies of the paintings and other artefacts pertaining historically to these rooms. Spatial augmented reality could, however, provide another solution [Bimber and Raskar 2005], where projections of virtualized objects would be superimposed over the real space and could be accompanied by animations and other elements capable of increasing people's involvement. In such a realization, the virtual information layer could be integrated almost seamlessly inside the real world, reducing the estrangement factor for non 3D-experts [Ridel et al. 2014]. Outside the Ducal Palace of Mantua, it is also possible to set up a simulated studiolo made of real bare walls, in plywood for example, and 3D printed or reconstructed objects, such as furniture or 3D printed copies of works of art, to be used as simulacra of the objects visualized inside the virtual world. Here, visitors would experience a situation of hyper reality, with a seamless intermixture of physical and virtual worlds [Tiffin and Terashima 2005]. Since the construction of projects such as the Museum of Pure Form [Loscos et al. 2004], where a complex haptic system was elaborated for giving users full and realistic interactions with sculptures inside a virtual environment, the realism of virtual reality experiences has improved exponentially. At present, environments such as The Void (https://www.thevoid.com/), a hyper reality environment for gaming defined by the company as “VR you can feel,” represent a possible frontier for Digital Cultural Heritage as well. Such a level of immersivity and “realism” may suit both scholarly and general audiences. 360 degrees of free movement combined with environmental audio would simulate a lifelike experience that opens new perspectives for researchers on topics that will be no longer isolated but rather clearly interconnected with other dimensions of research on Isabella’s life and milieu. A still further step may be that of setting up a collaborative virtual reality environment [Masson, Daffy, and Perlin 2017] [Berford et al. 2017] to create a working space that is compatible with a hyper reality experience.

Sentiment analysis techniques [Liu 2010], may enable forms of Big Data analysis to enhance even further our knowledge of the intimate social space of the studiolo. The capacity to extract participant sentiment automatically is important in today’s world of commercial marketing, while measuring public sentiment has become a primary tool for political analysis [Liu 2012]. Big data produced through the sophisticated algorithms of social media like Twitter and Facebook are the digital outcomes of these efforts [Pak and Paroubek 2010]. In the case of Isabella’s cultural patrimony, her voluminous collections (of letters, music, and works of art) may represent future digital corpora for sentiment analysis and opinion mining tools, and a challenge for the development of new Artificial Intelligence networks. Might new neural networks, developed and trained by interdisciplinary scholarly teams, inquire into Isabella’s digital, multimedia heritage in search of new perspectives on the Big Data concentrated in this minute, historic space?

Our challenge in the study of Isabella d’Este’s famous multimedia studiolo is to learn from a variety of data sources that are relatively sparse, in order to realize a multimodal dataset that can be experienced in a specific spatial context: the Virtual Studiolo. Multiple data sources within the project will complement each other, and their careful fusion will increase knowledge about Isabella d’Este, her practices as a collector, and the values of her time. As the Virtual Studiolo develops, we will focus on immersive technology through CAVE and museum-situated applications to add to this project the critical dimensions of scale and spatial proprioception. In both the desktop and the 3D immersive applications that will be our initial products, we attend to the textures and the aesthetics of this space, not only as principles of beauty but also in the shaping of a sensorium where Renaissance visitors affirmed their own values, desires, and emotions. We would welcome feedback from readers on the video or the project as a whole.

Following submission of our proposal (with video) in 2018, the Virtual Studiolo project was awarded a National Endowment for the Humanities Digital Advancement Grant, Level II, for 2019-2020. This generous support will enable construction of phases one and two of five proposed, foundational phases of the environment.

The authors wish to thank: Peter Assmann (Palazzo Ducale di Mantova); Anne MacNeil, Molly Bourne, Stephen Campbell, Daniela Ferrari, Lisa Regan (IDEA project); Luca Govoni, Micaela Spigarolo (Cineca); Francesca del Torre, Stefan Zeisler, Christian Mendez (Kunsthistorisches Museum Wien); Dominique Thiebaut (Musée du Louvre); and the Metropolitan Museum of Art, New York.

The realization of the concept-demonstration video was supported by the Samuel H. Kress Foundation, the University of California, Santa Cruz, and the Kunsthistorisches Museum Wien.