Dr Lora Markova, as part of a collaboration between CRAIC, UKRI and the V&A
Building on the success of the UKRI Audience of the Future (AotF) programme and in helping to secure its legacy, the Creative Research and Innovation Centre (CRAIC) is working with the UKRI team to convene a series of workshops on collecting and archiving immersive experiences in partnership with the V&A Research Institute (VARI). The workshops aim to define and scope the challenges of archiving immersive experiences with a view to creating a longer-term initiative that will bring together a wide range of creative and technology companies with archivists and researchers. The discussions will seek to propose ways to address the challenges for future collection, archiving and research and to generate ideas for pilot projects which could be undertaken to test methods for ‘experience-capture’.
Feeding into the V&A’s active involvement in Towards a National Collection (TaNC) project – using digital technologies for the creation of a unified virtual national collection across institutions – and drawing on a range of case studies and examples, including the AotF Demonstrator Programme, the discussions address a range of questions such as: What strategies are currently available for the preservation of immersive, VR, XR, AR and 360-degree experiences? How will immersive content remain accessible for future generations and audiences? How can cultural heritage and memory institutions standardise digital preservation formats, practices, and methods? How can creative technology companies, the museums sector, policy makers, academics and user communities develop collaborative methods for digital preservation?
This essay provides an overview of academic and museum-led research on the preservation of time-based, digital, and immersive experiences, and identifies a number of topics that might form the basis of further investigation.
Why archive and collect immersive experiences?
Contemporary cultural expressions and creative innovation within the Creative Industries – and increasingly so in the (post-)pandemic context – are being generated through the use of digital and spatial computing technologies, such as AR, VR, XR and haptics. Immersive experiences (through media arts, culture, and entertainment) form a crucial part of today’s creative and digital landscape whose cultural value and significance should be recognized as emerging heritage to be protected and preserved for current and future generations.
Archiving, collecting, and preserving ‘born-digital’ objects and immersive experiences pose challenges for contemporary conservation due to issues of technological obsolescence and intangibility, forms of collective and distributed ownership, and the interactive, participatory, and performative nature of digital and emerging technologies. Unlike traditional artworks and analogue objects, born-digital artefacts are ‘dynamic content’ as defined by the digital preservation expert David Rosenthal (2015). In this sense, practices of archiving, collecting, and preserving born-digital objects and experiences have transformed from avoiding to maintaining ‘inevitable change.’ (Depocas et al. 2003)
A range of museum-led initiatives have paved the way to an emerging field of digital preservation of immersive content and immersive experiences; for example, the preservation strategies and challenges tackled by the Variable Media Initiative (VMI), the Conserving Computer-Based Art initiative (CCBA), the Rhizome Initiative, and two recent research projects – Preserving Immersive Media (since 2018) carried out by Tate and Preserving and Sharing Born-digital and Hybrid Objects (2021/22) by the V&A.
Context – collection and preservation policies
UNESCO’s Charter for the Preservation of Digital Heritage (2003) defines digital heritage as the “resources of human knowledge or expression, whether cultural, educational, scientific and administrative, or embracing technical, legal, medical and other kinds of information, increasingly created digitally, or converted into digital form from existing analogue resources. Digital materials include texts, databases, still and moving images, audio, graphics, software, and web pages, among a wide and growing range of formats. They are frequently ephemeral, and require purposeful production, maintenance and management to be retained.” Digital preservation is then described as the process aimed at ensuring the continued accessibility of digital materials, which requires digital objects to be understood and managed at four levels: as physical phenomena; as logical encodings; as conceptual objects that have meaning to humans; and as sets of essential elements that must be preserved in order to offer future users the essence of the object. (UNESCO 2003) From a practice-led perspective, digital preservation has been defined as “the series of managed activities necessary to ensure continued access to digital materials for as long as necessary” by the Digital Preservation Coalition (2015) – a UK-based non-profit organization that advocates for digital preservation and long-term access to digital content and services.
UNESCO’s conceptualization of digital heritage and preservation tends to favour an object-based understanding that does not explicitly grasp the dynamic, interactive, distributed and networked nature of digital culture and creativity. This approach has implications and leaves conceptual and technological gaps in recognizing and preserving accelerating innovation and the value of the Creative Industries, and specifically emerging immersive experiences, as potential cultural heritage for future generations.
The V&A offers a leading example of the acquisition, display and preservation practices that have been closely related to the Creative Industries, as well as a long-standing practice of collecting performing arts. The V&A’s Theatre & Performance Archives, including the National Video Archive of Performance (NVAP) provide relevant directions for documenting the experiential, time-based and ephemeral features of creative immersive content. Moreover, the V&A’s Collection Development Policy (2019) and its six strategic objectives aim to “focus and deepen the relevance of our collections to the UK creative and knowledge economy” – documenting and making accessible design and creative processes and inspiring the work of practitioners across the Creative Industries. For instance, in terms of collecting and documenting information design, the V&A has acquired a small number of mobile apps since 2014. The collection approach adopted for the Chinese social media app WeChat is based on collaborating with the creators to secure a version of the app freed from all networked dependencies (a demo version) and populated with fabricated content that exemplifies typical user activities in order to avoid using real content subject to privacy and legal barriers.
As V&A’s recent Preserving and Sharing Born-digital and Hybrid Objects from and across the National Collection research report (2022) highlights further the extent to which the tech industry has become the gatekeeper of part of our cultural heritage – raising new questions around the intersections between media and cultural policy. The report signals a number of gaps in current policy and the need to develop more experimental and small-scale pilot initiatives to help understand needs and requirements.
Preserving born-digital objects and new media artworks
An early approach to the preservation of new media art (often born-digital and time-based) lies in the Variable Media Initiative (developed in 1998 by Jon Ippolito, an associate curator at the Guggenheim Museum in New York) and the Variable Media Questionnaire (VMQ), launched in June 2003 by the Daniel Langlois Foundation and the Guggenheim Museum. Through using an interactive questionnaire, the variable media paradigm maps creative intent and proposes artist-approved strategies for the preservation of artwork created in ephemeral and obsolete mediums. VMI employs four preservation strategies: storage – defined as preservation of the tangible or material, rather than the interactive or experiential elements of the work; migration – transferring the work into contemporary hardware or software; emulation – mimicking the original experience as closely as possible with the use of current and emerging technologies; and reinterpretation – recreating the behaviour of the work in a completely different medium according to a score or other interpretive instructions.
The Guggenheim’s collection also hosts the Conserving Computer-Based Art initiative (CCBA) which is an ongoing, practice-based research project that emerged as an attempt to address issues like hardware failure, rapid obsolescence, and the lack of established best practices in the preservation of computer-based art. CCBA argues that within documentation, conservators should analyse the work “to identify its vulnerabilities, for example, its dependence on a specific operating system, a specific piece of software, or customized hardware. Conservators must understand the technical dependencies of the artwork and must also identify and secure the proper resources for the long-term preservation (such as specific hardware and peripherals, source code, executable programmes, or media assets like video or audio files). These physical and digital components often vary depending on the technology employed in the artwork” (CCBA). In this sense, as defined by Deena Engel (2016), the CCBA initiative aims to respond to the challenges of “ongoing digital innovation and the resulting inherent instability of digital artworks.”
The V&A’s AHRC-funded project Preserving and Sharing Born-digital and Hybrid Objects goes beyond the realm of technical obsolescence. The recent report including research findings from the project discusses the multi-layered and complex authorship of many born-digital objects associated with communities or corporate ownership and expands on the potential of collaborative approaches to collection and preservation, suggesting industry and community involvement in the decision-making processes.
Preservation of immersive content and related challenges
Tate’s research project Preserving Immersive Media is a major investigation into the challenges of preserving immersive content and offers pragmatic recommendations to creative practitioners and collecting institutions. It explores the suitability of established digital preservation strategies for securing long-term access to immersive media. The project outlines four groups of strategies for the preservation of artworks employing immersive media such as 360 video, real-time 3D, VR, AR and MR, namely: stockpiling, hardware migration, emulation, and code migration. (Ensom and McConchie 2021).
Patricia Falcão and Tom Ensom from Tate’s Time-based Media Conservation team (a section of Tate’s Conservation Department) have previously investigated the main risks faced by digital objects – the loss of the bits of digital information and the loss of the means to access information in the context of ongoing technological change, which they have defined as content-centric and object-centric preservation challenges. In relation to software-based art and digital objects, Falcão and Ensom emphasize the relevance of emulation (and related techniques as virtualization) as strategies that can maintain and recreate specific technical features (e.g. lighting and processing of the rendering engine), as well as the ‘performative’ character of virtual environments – “the idea of integrity of performance runs through all software preservation work and achieving this could be considered one of the conservator’s main goals.” (Falcão and Ensom 2019) Another discussed approach is the focus on the code – “here emphasis shifts from understanding technical environment, to understanding what the software does and how this might be changed or reimplemented in order to avoid obsolescence.” (Falcão and Ensom 2019)
In the context of the dynamic content of VR environments, migration – as the transformation of an older digital file-format into a newer file-format – has emerged as a difficult and time-consuming method for cultural institutions that contradicts the mass digitisation efforts embedded in archive workflows, as argued by David Rosenthal (2015) and Candice Cranmer (2017). Hence, VR and other interactive content is thought to be “most legibly preserved” within the “inter-dependent software environments in which they were created,” as proposed by the Rhizome Initiative (Rhizome 2016).
Since not enough machine actionable metadata might be generated to allow for the effective matching of execution environments to artefacts, Rhizome has begun to develop an open-source software tool and dataset that could automatically connect collections of digital artefacts (including software) with emulation environments for re-enactment of the collection’s contents. As Cranmer has observed, “this would shift the focus of emulation from single objects into mass preservation, which would present a welcome change in archival practices.” (Cranmer 2017, p. 12)
Experimental approaches of emulation and virtualization include physical mapping of user interface, interaction modalities and locomotion, as explored by a recent research project at the University of New South Wales, Sydney, that proposes strategies to capture, simulate and reproduce experiences originally designed for large-scale immersive architectures within VR. The project provides a framework to capture the embodied aspect, the emotional engagement and the dimensional extend which are central to immersion. (Kuchelmeister, 2018) A comparable approach has been undertaken by the Karlsruhe Institute of Technology proposing a ‘synesthetic documentation’ method and e-Installation as a virtualization method that reproduces all synesthesia, interaction, and meaning levels of re-enacted artwork via advanced 3D modelling and telepresence technologies which offers an intermediate and long-term solution to archiving and dissemination. (Morcillo et al. 2016)
Capturing audience experiences
Audience experience has emerged as an important yet under-researched subject in the literature on contemporary conservation. William Real (2001) considers the documentation of time-based forms as “preserving for the future something that is above all an experience [which] might require conservators to take a more fluid view of what may or may not be changed about a work, challenging conventional notions of accuracy and authenticity.” Sherri Irvin (2006) suggests that preservation practices should be applied in correspondence with the experience that the author has designed and sanctioned for the audience. Similarly, Marga van Mechelen (2006) discusses the “ideal, intended experience” as a guidance for decision-making processes and preservation. Moreover, in relation to the preservation strategies of emulation and virtualization, David Rosenthal (2015) has introduced the notion of experiential fidelity to describe how close a recurring user experience can be kept to the original viewer’s experience.
Central initiatives on digital preservation, such as the VMI and the Capturing Unstable Media project have also recognized the importance of documenting audience experiences of new media objects. Practice-based approaches involve video recording of audience interactions to capture the experiential, spatial and performative nature of time-based works. Gabriella Giannachi’s 2012 project on documenting mixed reality performance develops tracking of participant trajectories as a method to capture audience-generated and audience-facing documentation and introduces an original archiving tool (CloudPad) that integrates ‘cloud computing’ into the annotation and synchronisation of mixed media resources. Deena Engel and Glenn Wharton (2014) also argue that the aesthetic experiences of users (or viewers) must be thoroughly documented and understood both through data models and source code documentation, as well as through non-technical user-point-of-view narrative-style documents.
Hence, it becomes relevant to investigate how collecting institutions and industry can address the collective and distributed ownership of digital objects, and the role of end users and user-generated content in practices of digital preservation. In this context, consideration needs to be given to several of the projects within UNBOXED: Creativity in the UK – the UK-wide festival of cross-sector and STEAM-induced creativity, offering a programme of ten experiments in creative technology. For instance, the stroboscopic-light spectacle Dreamachine by Collective Act invites audiences to document and revive visual and sonic impulses through drawing and retelling individual experiences – a practice that can be defined as a form of audience-generated archiving. From a different perspective, StoryTrails by StoryFutures ‘re-enacts’ communities’ oral histories through immersion into AR and VR experiences generated through 3D scanning and creating an archive of the present.
Towards industry-led archiving
Recent studies, such as Evanthia Samaras’ doctoral project on the preservation of visual effects in the screen industries (2021) proposes that archival practices should be adopted to assist the industry in self-archiving their records more effectively and to support industry recordkeeping and production business activities that facilitate the formation of archive collections. This can ensure that valuable evidence of visual effect production – records about the technologies, artistic process and working life of an important field of practitioners – and its continuing value that extends beyond production outputs is preserved over time for future generations.
Another recent example of a collaborative preservation approach is the partnership between BFI National Archive and the UK-based art and design studio Marshmallow Laser Feast in developing a pilot experimental acquisition of the VR project In the Eyes of the Animal. The case study “highlights the importance of experimental and research-led collecting practice to understand requirements, build capacity and redefine expectations for both institutions and creators.” (Preserving and Sharing Born-digital and Hybrid Objects report 2022).
Similarly, Finding in the Future of Live Performance provides a multidisciplinary ‘living archive’ that documents recorded talks and shares insights from the Audience of the Future project and learnings from multiple R&D projects run between 2019 and 2021 by the creative partners of the AotF Live Performance consortium. The Consortium, led by the Royal Shakespeare Company (RSC), is a group of 15 specialist organisations and pioneers in immersive technology bringing together their expertise in theatre and performance, the music industry, XR and virtual production, gaming, and the research sector to explore live performance using VR, AR, and MR technologies.
Thematic workshops
The first workshop session on live and real-time experiences (convened on 20 May 2022) showcased insights on the collaborative, creative, and technical aspects of making the RSC immersive production Dream (2021), such as building audience interaction systems that introduced real-time digital puppetry and real-time animation driven by live performance. The series of documented conversations and related catalogued documents published on the Finding in the Future of Live Performance site can be also positioned as an emerging example of industry-led archiving.
The following session (on 25 May 2022) on location-based experiences used as case studies the immersive R&D project Virtual Veronese developed by StoryFutures, in partnership with The National Gallery, and Curious Alice: The VR Experience that emphasizes V&A’s role as both collector and creator of immersive content. The discussion oriented towards generating a shared and in-depth understanding of the collaborative strategies and approaches that museums and CreaTech companies could develop towards archiving and collecting immersive experiences.
The final workshop (23 June 2022) on interactive and participatory experiences presents findings from the Digital Catapult’s The Creative Immersive Content Lifecycle: from Distribution to Restoration report (Jarvinen, 2022) with specific focus on existing and emerging technical documentation practices. In the context of the AoTF Demonstrator Programme, the discussion addresses questions as “will anyone be able to access and play – or research – the once widely available location-based AR game The Big Fix-Up from Fictioneers in two or three years or beyond? What documentation of it is left for conservation?” (Jarvinen 2022, p. 40) In this sense, the session aims to draw parallels between the challenges of archiving immersive experiences and the preservation of computer games as a related field due to games’ interactive characteristics and dependence on specific generations of microprocessors, display technologies and operating systems that face the challenges of technological obsolescence.
Overall, the Collecting and Archiving Immersive Experiences workshop project aims to outline the pathways towards establishing a distributed network of knowledge and collaboration between memory institutions and creative technology companies in sharing experience and best practices of archiving and collecting immersive content. The discussions generated through this project attempt to establish a common language and definitions, and a shared understanding of the cultural importance of archiving innovative creative expressions, as well as to match immersive modalities and technologies, e.g as classified in UKRI’s Creative Technologies Framework (2021) with corresponding established and experimental strategies of digital preservation. The project also aims to map the potential of museum collections to inspire creative practice and outline a trajectory towards industry-led and collaborative practices of digital preservation across collecting institutions and the Creative Industries. Further insights generated from the Collecting and Archiving Immersive Experiences workshops will be published at the end of June.
May 2022