DHQ: Digital Humanities Quarterly
2020
Volume 14 Number 3
Volume 14 Number 3
A Case Study Protocol for Meta-Research into Digital Practices in the Humanities
Abstract
This paper presents a multicase study protocol for meta-research in Digital Humanities, prepared by Digital Methods and Practices Observatory (DiMPO) Working Group of the Digital Infrastructure for the Arts and Humanities in Europe (DARIAH). The protocol is intended to help researchers in conducting meta-research and adopting this methodology for different purposes, disciplines and approaches. As many of the issues raised here are already covered in manuals for social research, our focus is the specificity of meta-research in the DH environment. The specificity of DH in this respect relies on an intrinsic challenge of bringing together generally undertheorised approaches of the humanities with very formal and process-driven ICT approaches.
The main assumption behind this research is that a meaningful change in scholarly practices is taking place and is worth investigating. Moreover, a full assessment of this transformation should not focus exclusively on pioneering research, but rather on the selective uptake of digital practices and methods by researchers in the humanities: those who do not necessarily affiliate themselves with DH but simply use digital tools to explore particular problems. Hence, this paper should be of interest not only to researchers willing to conduct meta analysis, but to all DH practitioners willing to gain critical perspective on their work, as well as for those working on funding, evaluation and research policy.
Three pilot studies are discussed in this paper, as they served as a basis for the protocol. They focused on different “units of inquiry” (individual researchers, projects, research communities) and varied in methodological directions: (a) individual interviews with Polish DH researchers; (b) mixed-methods analysis of digital practice in E-CURATORS, a multicase SSHRC Insight project focusing on archaeological research sites or projects, integrating individual interviews, document analysis and naturalistic observation; and, (c) group interviews with historians and literary scholars conducted within the framework of NEP4DISSENT COST Action. The resulting protocol is discussed in detail and some directions for further research are suggested.
Humanities work is changing.
A historian in Northern Europe uses Facebook, Youtube and other social media to study memory of dissent. She sees social media as a valuable resource for the study of memory shaped by public access and personal interactions.
A memory studies scholar from Bulgaria admits that the starting point for her research is often a term written into a Google search tab, which later develops as a ‘snowball’ helping to find essential websites for the study of cultural memory.
A historian of the early modern Ottoman court re-purposes a popular note-taking software to create a linked data system hierarchizing all of her archival notes. In her efforts to track the contact zones and cultural intersections of her subjects, she creates a valuable rich, structured dataset.
A media studies researcher who has spent over two years compiling a database of more than 9,000 articles on the Spanish Civil War in the Norwegian press, entirely manually coded, asks what text mining tools might help him further explore his materials. He also economises his time in the archive through the use of a tablet device, but keeps both microfilm and microfiche readers in his office to access some of his sources.
An archaeologist collaborates with first nations communities in the Arctic to record archaeological sites using smartphones and uses drone photography with digital 3D panoramas to recreate indigenous landscapes.
These examples, selected out of many stemming from our meta-research into digital
practices in the humanities (i.e. the study of research carried out with the help of
digital technologies), illustrate how technology is penetrating unevenly into different
stages of the research process: the discovery phase (search &
access to research data); gathering activities (data
collection); conceptual work and organization of one’s research assets (data curation), and computer-assisted analysis of the research
assets (data analysis). Invariably, these examples also
problematise the institutional, custodial, and disciplinary nature of humanities
research conducted within the confines of the physical archive, the geographically
bounded community of scholars, and the materially constrained toolset of a single
discipline. If scholarly research often now appears to transcend locality and place,
methodological approaches to studying how scholars work should also adapt
accordingly.
As we come to reflect on how humanities research is shaped today by the physical and
material factors that condition the work of researchers, as well as the new capabilities
that the digital ecosystem affords, we see that there are micro-practices — visible in
those examples described above — that fall outside both the traditional humanistic
methods and what is usually considered Digital Humanities (DH). We understand DH in the
broadest possible sense, as the “big tent”
[Svensson 2012] of digital practices, methods, tools and resources which
inform various disciplines in the humanities. Moreover, we do not limit its scope to the
core DH community of researchers advancing cutting-edge methodologies, but also look at
scholars who would never call themselves digital humanists yet do take advantage of the
digital to answer their non-digital research questions. “Big tent” DH recognizes
that the importance of the digital is not limited to the use of computational,
algorithmic methods for research, but has to do with the broader shift of the whole
paradigm of humanities proper: the centrality of digital resources and access to them
for all kinds of humanities work; changes in scholarly communication, peer review and
publication where digital and online systems are central, inter- and
cross-disciplinarity; as well as working with open content and open data, both digitised
and born digital.
In this way, we can see how research practices are fundamentally unpredictable: they do
not fall under the umbrella of a single custodial environment but are rather assembled
as a local response to a single project’s needs. To assemble them involves repeated acts
of bricolage, tinkering with moving parts that were not designed to work together.
Nowhere is the impact of this status more present than in the situatedness of humanities
research, that is in the (self-) management of its physical, virtual and mental spaces.
SPARKLE (Scholarly Primitives And Renewed Knowledge-Led Exchanges), one of the studies
we reference below, showed in particular the degree to which location and spatial
configuration were felt by the researchers interviewed to be integral to the
productivity of the humanistic scholar [Edmond et al. 2016]. We are perhaps used
to this aspect of, for example, archive-based work, due to the external constraints of
opening hours, reading rooms, and all the rest of what Arlette Farge refers to as their
“absurd rules of operation”
[Farge 2013, 52]. But what perhaps has its roots in a time-management
challenge progresses into what manifests as a highly individualised and expertly tuned
set of spatial strategies. Even in the relatively small SPARKLE sample, there was
essentially no agreement, of where the ‘best place’ to work might be. In part this was a
reflection of adaptation to different life circumstances: senior academics with teaching
and administrative responsibilities seemed generally to find their offices more rife
with distraction than their home spaces, for example. It also sometimes indicated a
separation, or multiple separations, between administrative work, personal life and
research work. But even these general rules had exceptions, where the desire for
work-life balance created yet further variation (libraries or coffee shops, for example,
or working in the office but out of business hours, on weekends or in the evening).
The nature of humanities research as being physically unbounded, or perhaps, more
precisely, loosely and alternatingly bound to multiple places (the office, the archive,
the classroom, indeed even the virtual environment or page of writing etc.) places it in
stark contrast to laboratory science, in which the nature of the work space is very
specific and determined, forming (at its most extreme extension) a guarantor of
scientific credibility, that is a “clean” or “controlled” environment. The
contrast between the closed physical space of the laboratory and nomadic nature of the
humanistic knowledge-creation process, with its constant refreshing of sources,
backdrops and inspirations, is reminiscent of the contrast between farmed agriculture
and the hunter gatherer, with the pre-agrarian mode of subsistence being perhaps more
dependent on shifting availability of resources in a wider space, but benefitting
instead from a more multisensory, kinetic and synthetic mode of cultivation.
The idea that objects should be organised in physical and mental space also
interconnects with another theme that appeared across our study, which is that knowledge
creation is an embodied process, encompassing far more than just the brain and its
focussed sense of sight as an organ for processing source inputs. This is no less true
in virtual than in physical spaces. Separation (of tasks, of ideas) and integration (of
source material, with a community) are constantly in balance in the epistemic process,
to the extent that one interviewee stressed the importance of technology for him in
making it possible to access his entire personal source library from his computer at all
times. These mental and physical environments, as well as the broader sensory world, of
those engaged in historical humanistic research leads to an enduring dominance of
multimodal personal research environments. In spite of the rise of digital humanities
(which some, but not all, of the interviewees had embraced), historians still prefer to
have notes, notebooks, pencils, post-its, books, printouts, and all manner of stimuli
around them, in addition to their laptops and digital tools. This set of hybrid
practices, spaces and tools are therefore customised to the questions and the knowledge
base of the researcher(s) using them, and yet must also be flexible enough to
accommodate research teams that evolve over time, software that is continually
versioning up, and workflows that must be reinvented with every new experiment, even as
they expand to incorporate new technological affordances. How can we study this kind of
subtle and evolving phenomenon?
This paper proposes an approach to qualitative meta-research into contemporary
scholarship which enables us to track and understand novel research practices and assess
their impact on scholarly activity as a whole. Our approach is based on several pilot
studies, and our goal has been to use these studies as “ground truth” to support a
theoretically-informed, highly original methodological framework for qualitative
research on DH practice: a self-reflexive approach to DH practice which is often
overlooked in existing empirical studies. The primary audience of this paper, given its
methodological scope, are DH researchers willing to either conduct meta-research, or to
critically evaluate their own work. We believe that this kind of research should be of
interest to researchers willing to understand their own method and be empowered to be
critical about their work (in particular people embarking on large-scale collaborative
projects). Additionally, anyone interested in interdisciplinarity should find much to
think about here as well. Beyond the academic, anyone looking to fund, evaluate, promote
or integrate DH (whether as a Dean or funder or research manager or policy maker) could
make use of this perspective.
The rationale for meta-research in DH, viewed as an integrative framework, could be
explained in four dimensions: ontology (how do we conceptualize the domain of research),
epistemology (what is the knowledge we are seeking to produce), axiology (what values
shape the object and process of our research, i.e., the infamous “so what”
question), and methodology (research design and method selection to address these
issues).
At the ontological level we conceive of DH research as a practice based on symbolic
interactions involving purposeful action by researchers. Hence DH is treated not merely
as empirical reality (accessible only through empirical study) but also as consisting of
such elements as motivations and norms of researchers, or goals and objectives served by
their research choices, or the criteria, often implicit, of what constitutes “good
research.” These ontological commitments bring our work close to the model of
purposeful human activity established by cultural-historical activity theory [Leont’ev 1978]
[Engeström 2000]. Figure 1 offers an informal model representing this
ontological view.
As for epistemology, our approach relies on access to diverse evidence drawn from
interviews with actors, but also from examination of documents, records and other
traces. At the same time, building from the ontological assumptions, our protocol does
focus on understanding what motivates and what shapes DH researchers' work, i.e. it
seeks to prioritize objects of our study from the viewpoint of research
participants.
As for the axiological level, we advocate for DH researchers (not just meta-researchers)
to engage in what Pierre Bourdieu calls “epistemic
reflexivity”
[Bourdieu and Wacquant 1992]. Central to Bourdieu's idea is the notion of “two steps
back” in research: the first step back is to look at the research field itself. In
other words, we “zoom out,” to use Nicolini's
(2010) different formulation, to view more clearly the main entities and
processes-relations in the field of study. The second step back is to look from the
outside at the approach, methods, and instruments of research (i.e., for a DH
researcher, to consider the “epistemic framework” or commitments of their own
research, or the research of others DHers). This kind of “double objectification”
is what we suggest through our call for meta-research.
Finally, our methodological plane coheres with the commitments of the study detailed
above. We employ our multi-case studies meta-method within a critical realist grounded
theory, i.e. we neither adopt the notion that we start tabula rasa seeking to "hear" the
data speak to us, nor do we believe that it is enough to just start with hunches and
“sensitizing concepts” and take it from there.
In some ways our meta-research on DH can be equated to the kind of methodological work
that goes on in any community of practice (bioinformatics, for example), or indeed is
akin to the work of Science and Technology Studies (STS) and the Sociology of Scientific
Knowledge (SSK), but there are a few key differences. Primary among them would be the
gap in epistemic vocabulary and different levels of methodological coherence between the
diverse fields we examine. Unlike the relative coherence of work within specific
sciences that was noted in STS/SSK studies, e.g. the existence of coherent epistemic
cultures in molecular biology and in high energy physics identified by Knorr-Cetina (1999), humanities disciplines tend to
be less coherent methodologically, especially when contrasted with information and
computer sciences. In other words, humanities are generally undertheorised in terms of
their methods and epistemic strategies, while software development seems at times
perhaps overly process-driven in this respect. There is a distinct challenge in bringing
these approaches together. In addition, the starting point for this work has not been
entirely disinterested: this is an investigation of DH, but also performed — to some
extent — by those trained primarily as (digital) humanists, not as social scientists.
Because of this, even when we use social science tools and instruments, we may use them
differently, drawing on the strengths of what we were taught to do (read texts, looks
for patterns, use a peripheral view, be sensitive to subtle, trace indicators of bias or
habitus, be unwilling to accept that language use is neutral, focus on cultural norms,
values etc). The resulting positioning of our approach to understanding how DH research
works, brings certain restrictions on its scalability or universality. While it would be
possible for observers of other inter-disciplines and methodological communities to take
inspiration from the manner in which we have leveraged our perspective on the work of
those in our immediate environment, the specificities of how that would optimally be
deployed could be quite different and lie beyond the scope of this paper. That said, our
approach would be quite extensible to most DH research and researchers seeking a
framework from which to observe their fields.
In what follows, we present a review of previous studies on digital practices in the
humanities, before moving on to discuss the steps taken by the Digital Methods and
Practices Observatory (DiMPO) Working Group of the Digital Infrastructure for the Arts
and Humanities in Europe (DARIAH) to develop a protocol for DH meta-research. Included
in that section is a critical overview of the notion of meta-research, as it applies to
the DH field, and an account of earlier DiMPO work conducted with a meta-research
orientation. The pilot studies that follow point to promising directions in the area of
DH meta-research and lay the groundwork for the development of our protocol which is
presented in the penultimate section of the article. The DiMPO research protocol for DH
meta-research consists of a set of specific methodological guidelines for researchers
wishing to study research practices in the humanities. It includes: (a) the proposed
research workflow (summarized in Figure 3); (b) a conceptual model accounting for both
external and internal dimensions of DH activity; (c) a core code system (taxonomy of
descriptors) amenable to the qualitative data analysis (QDA), which could be expressed
in XML as a SKOS structure if necessary; (d) a qualitative interviewing and document
analysis codebook identifying themes for eliciting evidence in situ (elaborated in Table
1); (e) an exemplary case study structure to shape analysis and writing-up. The final
section sets forth the next steps for the protocol and the DiMPO working group.
We believe that broader adoption of this protocol by the research community could form
the basis for the improved exchange of qualitative research results in this area, and a
better understanding of DH and its penetration beyond the scope of those already
committed to these methods. In establishing the protocol we hope to illustrate diversity
and contextual dependency across different disciplines, schools of thought, typologies
of research project and team, and individual practices. Taken together, the article and
protocol map out an approach to meta-research of digital practices in the humanities
which goes much further than the state of the art, in that it asks us to consider why
and how we can comprehensively chart the impact of the digital on our knowledge
production practice.
Earlier Studies on Digital Practices in the Humanities
Considerable international research has been conducted on digital scholarship,
digital research infrastructures, the organizational and disciplinary structures
enabling digital work in the field of humanities research, and the information
seeking and use patterns of researchers in the human sciences (for an overview, see
Borgman 2007, Case and Given 2016, 287–96, Schreibman et al., 2016). Within information science, researchers in the
subfield of information behaviour have been trying to understand research practice as
far back as the 1950s. While this work was initially limited to information seeking –
how scholars sought information relevant to their research – it gradually expanded to
encompass processes of information management, curation, collaboration and
communication [Julien and Duggan 2000]. Information behaviour studies addressed
questions of computer use, comparison between the humanities and other disciplines,
resource discovery, primary and secondary sources, and finding aids and archival
services used by humanities researchers [Dalton and Charnigo 2004]
[Delgadillo and Lynch 1999]
[Duff and Johnson 2002]
[Stone 1982].
One strand of earlier information behaviour research in particular remains relevant
today: the abstraction of processes of scholarly information work, such as chaining,
browsing and extracting, as described by Ellis in his comparative study across the
sciences, social sciences, and the humanities (1993). This comparative approach was
enriched with further processes [Meho and Tibbo 2003], and later converged with
the notion of scholarly primitives introduced by John
Unsworth (2000). Carole Palmer and her colleagues further developed the
notion of scholarly primitives into a fully-fledged classification of twenty
granular, standardised and recombinant scholarly activities, within broader processes
of searching, collecting, reading, writing, and collaborating [Palmer et al. 2009]. On a parallel trajectory, the work of the Arts and
Humanities Data Service (AHDS) in the 2000s took the abstraction of the research
process further by establishing the complementary notion of computational methods.
This led to the development of an extensive taxonomy of methods for the arts and
humanities, documenting the context, needs and scenarios of use of each method in
particular disciplines – from history, performing arts and archaeology to theatre and
linguistics – across complementary contexts of digital data creation, processing and
use [Hughes 2008]
[Reimer 2009]. The AHDS methods taxonomy has been used to structure the
ICT Guides database of digital arts and humanities projects in the UK [Grindley 2007a]
[Grindley 2007b] (cf. 2006a, 2006b), and governed the information architecture of
the arts-humanities.net portal, hosted for several years by the Centre for e-Research
(CeRch, King's College London). Both the repository and the taxonomy were furthered
by the DH@Oxford initiative in the UK, and the Database of Research and Projects in
Ireland (DRAPIer, Digital Humanities Observatory).
The Digital Research Tools (DiRT) directory originally appeared under the auspices of
the Bamboo project [Loesch 2013], and created a different
classification of digital tools for humanities research which, implicitly, reflected
an underlying categorization of scholarly research activities. An initiative within
DARIAH-DE (the German tier of the DARIAH infrastructure) drew from such earlier work
in the pragmatic context of building a tagging scheme for a Zotero bibliography on
“Doing Digital Humanities”[1]
to develop an updated Taxonomy of Digital Research Activities in the Humanities
(TaDiRAH) in collaboration with the DiRT directory [Borek et al. 2016]
[Perkins et al. 2014]. Echoing Ellis and Unsworth, DH researchers involved in
the development of methods taxonomies turned to modular, standardized procedures
underlying humanities research enabled by ICT, and usable in hugely diverse contexts.
One driving idea was to advocate for the development of “a system
that would span disciplines and facilitate knowledge transfer around digital
methods, could help prevent the re-inventing of the wheel, encourage re-use of
resources and contribute to a greater awareness of the importance of digital
research”
[Reimer 2009].
In spite of all of this work, however, our knowledge of the information behaviour of
humanities researchers in the digital environment remains incomplete, largely due to
the fact that this remains a diverse field and has been a moving target, especially
as the tools of scholarship have changed radically with the advent of ICT. The
complexity and fluidity of the research environment calls for flexible methodologies,
capable of capturing the nuances and accounting for the variety of research
practices. Hence, the emergence of qualitative inquiry approaches towards DH, which
tend to investigate digital practices in a broader epistemic and organizational
context, very much in-line with the methodologies presented in the following sections
(e.g., Antonijević 2015).
There are a number of recent studies which either directly or indirectly address
digital research practices in the humanities. An important subset of this research
has been conducted in the context of defining user requirements for digital research
infrastructures and services for the arts and humanities. Questionnaire survey and
qualitative interviewing research in the context of the Preparing DARIAH and European
Holocaust Research Infrastructure project [Benardou et al. 2010] highlighted,
among other findings: the persistent use of traditional, non-digital formats to
access textual archival resources and books; the high importance attributed to
collecting and managing references, as well as to storing both digital and paper
copies of both published and unpublished materials; the perceived value of
highlighting relevant text passages and storing notes with them; and the importance
of named entities for content-based retrieval of primary and secondary sources.
Further analysis of interviews under the same project formally corroborated a number
of important intuitions, such as a widespread tendency of researchers to use primary
data and secondary sources at the same time, or to forge links between objects on the
basis of their conceptual content [Benardou et al. 2013]. The Scholarly
Research Activity Model, resulting from this work, drew from mixed methods research
on the scholarly practices of researchers in the DARIAH and EHRI communities and was
used to model, represent and analyse the findings of such analysis [Benardou et al. 2010]. Some recent work within the DARIAH Community Engagement
Working Group (CE-DARIAH) continues to analyse the work practices of humanities
researchers and how these researchers might be encouraged to better engage with
Research Infrastructures (RIs) such as DARIAH [Garnett and Papaki 2019].
Likewise, numerous projects have tackled research on DH from different perspectives.
The CENDARI Project (2012-2016) was funded by the European Union’s Seventh Framework
Programme to pilot the implementation of a virtual research infrastructure for
scholars of medieval and modern history. A part of the project dedicated to the
Domain Use Cases aimed to capture the different research practices in transnational
history, challenges posed by the fragmentation of archival sources, and how a virtual
research infrastructure can help address these challenges (CENDARI WP4 team, 2013).
The DESIR project (INFRADEV-03-2016-2017; 2017-2019), created a very informative
series of video interviews with DH researchers, focusing in part on their own career
paths.[2] The
KPLEX project (2017-2018), funded under the European Commission’s Horizon 2020
research programme, conducted surveys and interviews on how big data research might
be better informed by humanities research practices. The most relevant pieces of work
for DH meta-research are the interviews of WP2, which looked at computer scientists’
attitudes toward the definitions of certain key terms in their work (especially
“data”), provenance and data cleaning as productive or destructive processes,
and the role of uncertainty in humanities research datasets. The protocol used,
background information and analysis of the interviews were published as a
report.[3]
The PROVIDEDH Project (CHIST-ERA, 2017-2020) prepared User Stories and Scenarios
which present eight cases of how uncertainty arises and is managed in early modern
historical research, drawn from interviews with four expert researchers in the field.
Finally, The Scholarly Primitives and Renewed Knowledge-Led Exchanges Project [Edmond et al. 2016] was inspired by ethnographic methods, in looking at the
practices of knowledge creation in the humanities. Nine interviews were carried out
with the aim of pinpointing the key moments and milestones of this process [Edmond 2018]. What we can extract from this distillation of almost three
decades of research in digitally-enabled humanities scholarship is that any broad
framework for study has to be expansive and customizable for a field-specific study.
It needs to account for the specificity of the sources, the research team and funding
structure, versioning software — in short, the radical instability of the rapidly
changing research environment as well as knowledge production itself. Even a cursory
examination of how researchers work with digital technology reveals that the
application of specific research methods is dependent on the researcher’s ability to
use particular digital tools and services (as well as on the stage in research
lifecycle, the disciplinary context, etc.). Those studies form the background for the
investigation on the scholarly practices, digital needs and attitudes of European
researchers in the human sciences which is the focus of the present discussion, and
of the meta-research approach that has been developed within DARIAH DiMPO Working
Group.
Towards a Protocol for DH Meta-Research
The number of studies presented in the previous section brings to the fore how
complexity is inherent not only in the human cultural record that is the object of
humanistic inquiry but also in the very practices that constitute scholarly work. In
the last five years, the work of DiMPO has aimed at studying the purposeful activity
of humanities researchers as they engage with everyday processes of scholarly work
and to document and account for the role of the digital in their workflows. Its first
research endeavour was the European survey on scholarly practices and digital needs
in the arts and humanities, disseminated in ten languages during winter 2014/2015,
which attracted 2,177 respondents from sixteen European countries. The survey aimed
at providing “an evidence-based outlook of scholarly practices,
needs and attitudes of European humanities researchers towards digital resources,
methods and tools across space and time”
[Dallas et al. 2017].
The DiMPO survey results depicted a landscape in flux, with rapid change in some
places and contexts, and surprising stagnation in others. To capture this changing
landscape, the survey was designed as a longitudinal research project, to be repeated
every few years. However, while a survey is a good instrument in identifying “what”
questions through quantitative summaries and statistical associations between
different variables, it provides little context to help elucidate holistically the
“how” and “why” of DH research. For this reason, DiMPO agreed that a
process of qualitative research is also necessary to allow a fuller understanding of
how humanities researchers work while engaging increasingly with digital resources,
tools and methods. Such a qualitative approach would complement the quantitative
while allowing for the interpretations to the multiple intentionalities, disciplinary
affiliations and contextual dependencies of particular developments in the field.
Before the next iteration of the survey, DiMPO initially adopted a case-study
approach, inspired by a multicase study model [Stake 2013], for the
following reasons:
- it would provide an insight into researcher’s motivations and their reasons for choosing (or refusing) certain digital methods while providing a perspective on the epistemological influence of tools on the research process;
- it would be better suited to capture and understand the moment of digital transition in the humanities, especially how earlier methodological or theoretical assumptions are negotiated when adopting digital methods;
- it might provide some hints on the evaluation of research work and new methods in DH;
- it might shed more light on who are digital humanists, or, more precisely, what it takes for researchers to consider themselves digital humanists;
- it accounts for the holistic, contextual and situational dimensions of human culture
Discussions of the rationale and priorities that should govern research on DH
research practice led DiMPO to consider a broader epistemic framework for its
activities, based on the notion of meta-research. Stemming from concerns on
reproducibility and waste in the biomedical field, meta-research emerged recently as
an important orientation to the study of research practice. While the term was
originally used as a mere synonym to meta-analysis, i.e. the consolidation of
aggregated findings of published studies such as different clinical trials on the
same research hypothesis, the emerging new field of meta-research now seeks to find
evidence-based improvements to scientific work through “research
on research”
[Wikipedia 2018] by looking more broadly at “thematic areas of methods, reporting, reproducibility, evaluation, and
incentives,” combining theoretical and empirical investigation, and
extending beyond retrospective studies to planned, prospective inquiry on scientific
practice [Enserink 2018]
[Ioannidis et al. 2015]. DiMPO shares the pragmatic goal of meta-research, as
it has been adopted in the natural and hard sciences, to make a positive difference
in the way researchers work. This commitment underlies the mixed methods research it
adopted, consisting of a combination of longitudinal questionnaire survey, literature
review, and qualitative multicase study investigations. The scope of DiMPO work,
driving the construction of the DiMPO protocol for case study meta-research, is
therefore adjusted to studying DH scholarly work by zooming in to a granular study of
particular research activities and operations and zooming out to considering broader
socio-technical and cultural factors. Beside contribution to knowledge, an important
objective of DH meta-research is to assess and help overcome challenges currently
faced by humanities researchers in creating, curating and using digital resources,
taking up and building useful digital tools and services, and developing and adopting
appropriate digital research methods in practice.
During a series of working group meetings in 2017-18, DiMPO planned for harmonization
between three pilot studies conducted by different group members and discussed their
outcomes, paving the way for the preparation of the final version of the DiMPO
protocol. These pilot studies were not designed for the purpose of this meta-research
endeavour: they rather represent active, actual research on DH scholarly practices
and knowledge work conducted by different co-authors of this study. These span
different disciplines and fields within the “big tent” of DH, from literary
studies to history and archaeology, different units for inquiry from individual
researchers to research projects and whole fields, and diverse kinds of research
questions from digital literacy to the effect of digital tools on research activity
to the formation of research communities. Collectively, and through their
complementarity and diversity, these pilot studies were selected to satisfy, through
the principle of saturation typical of qualitative research, a criterion of
generality for our proposed methodology.
Based on input from these pilot studies, a preliminary case study structure was
produced, including a common template for outcomes and reporting, instructions on
construction of each case study dossier and how it should be situated in the context
of multiple case studies. This framework was elaborated according to the particular
research needs of the pilot studies. Three studies, presented in brief in the next
section, account for three complementary research perspectives adopted by DiMPO,
focusing on a different level of “unit of inquiry” and adopting methodological
directions from this orientation: (a) individual interviews
with Polish DH researchers; (b) mixed-methods analysis of
digital practice in archaeological research sites or projects, integrating individual
interviews, document analysis and naturalistic observation, and, (c) group interviews with historians and literary scholars
conducted within the framework of NEP4DISSENT COST Action. A secondary dimension, on
which the pilots contribute differently, was to balance between a “grounded
theory,” inductive approach based on open coding, and a theory-laden,
conceptual model based approach based on practice studies, activity theory etc.
Finally, the choice of case-studies was motivated by the need to gather input on
digital practices in multiple disciplines (literary studies, history, archaeology),
reflecting our broad understanding of the humanities and the “big-tent”
characteristics of DH. In all, a total of twenty-seven researchers were included as
study subjects in these pilot studies. The following section briefly contextualises
the presentation of the protocol that follows and its main purpose is to give the
overview of research questions, methods and possible outputs that can stem out of
meta-research.
Pilot Study 1: Individual Researchers
This pilot study involved Polish humanities scholars and aimed to gain a better
understanding of the DiMPO survey results with regard to how digital practices change
the work of researchers in various fields, and how these researchers define and
understand digital practices in the context of their needs. It was assumed that a
series of interviews would provide better information on the values assigned to
digital technologies in the context of their use in research, to identify
particularly useful existing and missing digital tools, as well as potential
obstacles. Other research questions included the influence of digital tools on
communication between researchers and disciplines, dissemination of results, the
perception of what constitutes research data, and researchers’ assessments of the
level of digital competence among their peers.
The pilot study employed a relational approach to digital literacy, which defines the
competent use of the digital in terms of a contribution to one’s life quality and
performance [Filiciak et al. 2013]. This relative perspective allows us to
view digital competence in relation to other spheres of activity, allowing a
subjective assessment of respondents’ competences with regard to their actual needs.
In order to fruitfully explore the subject in the relational approach, the research
team employed the technique of episodic interviews [Flick 1997], as it
encourages the exploration of both semantic and episodic dimensions knowledge. The
former entails internalised social knowledge, whereas the latter expresses its
particular expressions based on specific actions. The method provided insights not
only into interviewees’ assumptions on digital research technologies and values
ascribed to them, but also into particular manifestations of these assumptions and
values in research.
The interview questionnaire consisted of three key sections, each containing up to 9
questions. Section one, aimed at eliciting semantic knowledge, included
general questions about the usefulness of digital technologies in research, along
with introductory questions identifying the interviewee’s main research interests,
career focus and current work situation. Section two concentrated on the
episodic knowledge, which regards to the following dimensions:
communication within the scientific community, disseminating research results, doing
research, academic teaching. The interviewees were very willing to share their
experiences with particular devices and software with regard to those dimensions.
Section three was dedicated to consolidating and juxtaposing both dimensions of
knowledge. The questions focused on the interviewees assessment of their individual
digital skills as well as the skills of their scholarly community. This section also
assessed the perceived effectiveness of using digital technologies in research in the
context of the non-digital practices.
There were five interviewees in total. The subject group consisted of one female and
four male researchers, representing different career stages (three professors and two
PhDs) and disciplines (two historians and three literary scholars). Two interviewees
were leading grant projects at the time of the interview. Three interviews were
conducted face-to-face in Warsaw, and two remotely via Skype. Each interview lasted
between 45 and 65 minutes. The interviews were recorded, live transcribed and coded
in vivo by the team members, i.e. new codes were generated
inductively from interviewees’ statements by two researchers separately listening to
the recordings and taking notes. To decrease the bias, the material was
cross-analysed by two researchers. Although this was manageable in the pilot study
due to the volume of material (five recordings, 60 minutes each on average), for a
full-fledged study full transcription and qualitative coding would be advised.
The study enabled investigators to answer the majority of research questions, and
proved effective in eliciting both the general attitudes and experiences concerning
the use of digital technology in research, and the experiences and assumptions
connected to particular instances of using such technologies on the various stages of
the research process. This short discussion of the study outcomes is by no means
comprehensive. Two key conclusions from this research were that digital practices
enable, facilitate, and in some cases demand the emergence of networks of cooperation
within the academia (i.e. researchers from various fields), as well as between
academia and other groups (amateur science enthusiasts for instance). Furthermore,
digital technologies and practices can help bridge the gap between academic centres
and peripheries (although limited access to digital technologies may have a negative
impact on this process), as it facilitates cooperation between units, opens access to
various resources, and offers state-of-the-art solutions to methodological or
theoretical issues.
As for more detailed conclusions, interviewees reported a major qualitative and
quantitative change with regard to possible research avenues, quoting perspectives
enabled by big data to conduct large-scale comparative studies. Digital research is
perceived as more cost- and time-efficient, facilitating the coordination of various
elements of the research process. However, the interviewees highlighted the need for
the conscious use of the new tools, which may give great opportunities but should
always serve as a means for answering the research question, so that the mere use of
tools on new material does not become a goal in itself. The writing was reported
least affected by digital technologies, as standard, widely available word processors
were not replaced in the studied sample by new tools. Interestingly, what was further
confirmed in the community research described below, researchers incorporate into
their work the non-scientific resources as, for instance, data provided by amateur
genealogists or individual enthusiasts. As for the physical working space,
interviewees tend to work remotely in different settings (home, university, library),
using mostly their own equipment.
In sum, this pilot study informed our thinking about the DiMPO protocol in terms of
introducing the methodological orientation of the episodic interview approach, and of
defining the research questions and questionnaire construction, and the workflow for
the individual interviews.
Pilot Study 2: Research Projects and Sites as Activity Systems
The second pilot study was conducted as part of E-CURATORS, a research project funded
by an Insight grant from the Social Sciences and Humanities Research Council of
Canada, seeking to advance scholarly knowledge on the growing complexity and
diversity of digital research practices in archaeology in the context of increasingly
mainstreamed, pervasive, and invisible digital infrastructures. The study looks at
practices such as: adoption of mobile devices to capture and document excavation and
survey data; use of off-the-shelf mobile apps to construct three-dimensional models
of archaeological artefacts, or to geo-locate archaeological information resources;
instant online aggregation of captured data and resources in research archives,
databases and repositories at the time of capture; use of synchronous and
asynchronous communication technologies to connect researchers with data and enable
interpretation “at the trowel’s edge”
[Hodder 1997]; collaborative annotation, enrichment and interpretation
of archaeological data using Web 2.0 technologies, crowdsourcing and social tagging;
adoption of virtual reality (VR) and augmented reality (AR) visualization methods and
equipment imported from fields such as the media and gaming industries for the
generation and assessment of archaeological knowledge; use of blogs, wikis and social
media networks to co-create and co-curate archaeological information objects; open
access provision of data outside established archaeological data infrastructures;
and, the use of gamification, storytelling and social media networks for public
communication, learning and mediation.
Inspired by activity theory [Allen et al. 2011]
[Engeström 2000]
[Leont’ev 1978] and its application to digital curation [Dallas 2007]
[Kaptelinin and Nardi 2006], E-CURATORS focuses on studying archaeological
research in the situated framework of bounded research processes considered as
activity systems, which involve a group of individuals purposefully
engaging in specific, hierarchically organized, activities prompted by multiple
motivations, pointing to specific goals, and made possible through the affordances of
a diversity of norms, established procedures, methods, competencies, as well as
digital and conceptual mediating tools. E-CURATORS is thus construed as a multicase
study on archaeological projects, organisational units performing situated
archaeological work, or sites of archaeological activity. Following a mixed methods
approach, for each case study, the project collects a combination of three kinds of
research data: (a) audio or video recorded testimonies and stories, acquired through
a combination of life history and semi-open episodic interviews, and covering
historical (factual), attitudinal and normative dimensions of archaeological
activity; (b) documentary evidence, including captured field data, digital photos,
models, notes, recordings, logs, database entries, maps, cataloguing texts, reports,
web content and publications, and (c), optionally, naturalistic observation of study
subjects. While the focus is on the overall activity system constituted by
interacting actors, objects and mediating tools, E-CURATORS aims to examine different
levels of granularity of scholarly activity through a process of
zooming-in to the specifics of digitally-mediated actions and
operations, and zooming-out to the contextual factors of sociotechnical
infrastructures and epistemic culture [Nicolini 2013].
E-CURATORS adopted a multi-step research activity design, based on connecting three
interrelated processes: conceptual modeling of digital archaeological practice, data
constitution, and analysis and theory building (these were integrated into the DiMPO
protocol, which is presented in more detail in Figure 2).
Work in the E-CURATORS project involves frequent iterations between activities, as
well as interaction between activities executed in tandem across processes. So far,
the E-CURATORS team developed a preliminary, extensible model and “code system”
for QDA, drawing from an earlier model on archaeological digital curation [Dallas 2015], and based on integration between relevant parts of the
NeDiMAH Methods Ontology [Hughes et al. 2016]
[Pertsas and Constantopoulos 2017], the CIDOC Conceptual Reference Model [Le Boeuf 2015], and the gIBIS practical argumentation and design
rationale schema [Conklin and Begeman 1987]
[Shum et al. 2006]. The model aims to represent evidence not only on (a)
activities, actors, their goals and motives, archaeological entities, information
resources, methods, procedures, digital devices, tools and services, but also on (b)
issues (problems, questions), positions (solutions, ideas), and arguments (criteria,
pros and cons) elicited from study subjects on their archaeological practice. A
preliminary schematic representation (Figure 1) identifies the functional structure
of digital archaeological practice through three overlapping facets, or views: (a)
intentionality, relating activities with research actors, beliefs they hold,
collectivities they belong to and shared drivers such as motives, goals and norms;
(b) process, relating activities with procedures guided by drivers, tools
implementing them, and information objects involved in them; and, (c) knowledge work,
relating activities with archaeological entities, information objects, and beliefs
they support or challenge (Figure 1). The code system consists of extensible code
stub hierarchies for all entities in the model, and the relationships between
them.
Figure 1.
E-CURATORS schematic model of archaeological digital research practice based on
activity theory [Leont’ev 1978]
[Engeström 1999] and scholarly activity models (SRAM) [Benardou et al. 2010]. The model represents archaeological activity though
actions such as surveying, recording, and classifying, performed by a diversity of
actors (e.g., excavators, finds specialists), who hold, establish or adopt claims
(positions, arguments) on archaeological questions through knowledge interaction
with archaeological entities (artefacts, ecofacts, sites, ancient people, events,
cultural meanings etc.) as data and information resources accessed by means of
diverse (physical, digital, conceptual) mediating tools (methods, procedures,
workflows, recording devices, and analytical software).Fieldwork on the first case study, the Archaeology Data Service (University of York,
UK) and its Internet Archaeology journal, was completed
in December 2018. Interviews conducted with seven key members of the ADS team are
currently being transcribed, coded and annotated using the MAXQDA computer-assisted
qualitative data analysis software (CAQDAS) [Woolf and Silver 2017]. In tandem, a
Neo4J property graph database [Vukotic 2015] is being developed to host
representations of research data (coded transcripts of interviews, annotated and
categorised documentary data), so that they can be visualised and further analyzed
through graph browsing, filtering, query, summarization and synthetic views of main
activities and ideas expressed by study participants, and related motives, goals,
norms and competencies, as well as digital infrastructures and tools, and
archaeological entities related to research practices.
An initial evaluation of results from this single-sited case study demonstrates
themes and underlying factors, such as shared norms and metaphors about digital
preservation work, the nature of the archaeological record, the role of digital
repositories such as ADS, and the value of specific knowledges and capabilities,
across study participants. Positional roles and relationships between researchers,
trans- and interdisciplinary interactions and “translations”, emerged from
interviews with different team members and documentary evidence they pointed us to,
and helps build an account of digitally-enabled archaeological activity at ADS that
would be impossible without attention to the collective work of study participants
within multi-year projects, and to the set of arrangements, presuppositions, methods
and norms shared by the ADS team. Individual competencies and orientations play
nevertheless as important a role as the broader context of “client”
organisations and people in British archaeology, peer organisations and networks
active in archaeological data archiving in the European or global context, and, last
but not least, a transdisciplinary ecosystem of diverse platforms, research
organisations in computer science and informatics, policy makers and funders whose
activities affect, directly or indirectly, the work of the organisation. Sensitizing
concepts such as objectual practice [Knorr-Cetina 2001], thought
collective [Fleck 1979], and semiosphere [Lotman 2005]
seem to be promising in providing fruitful theoretical insights on this study.
The ADS case study raises particular questions regarding the locatedness of
archaeological curation practice. ADS activity is predicated upon localised primary
archaeological fieldwork, but, by virtue of how communications between archivists and
“client” archaeologists unfold, and how information objects (such as
datasets, and archaeological reports) are produced and curated, the ADS work itself
detached from the location where it takes place: ADS staff work mostly in front of
computer screens, using information systems and services accessible via digital
networks, and interact with stakeholders over the phone, email and other
digitally-enabled means of communication. Viewed as an archaeological semiosphere
[Laužikas et al. 2018], the field where ADS research, curation and data
management work is enacted may be conceived as a (conceptual) landscape punctuated by
diverse disciplinary traditions, data management methods and rules-of-thumb, norms of
best practice and acceptance criteria, competencies and skills, systems and services,
and data standards, among other things. While place is not one of the entities
represented explicitly in the E-CURATORS schematic model (fig. 1), a topological
structure is implicit in its overarching conceptual organisation. Besides, the Neo4J
graph database was selected to represent E-CURATORS data exactly because of is
ability to allow visualisation, navigation and traversal of topological structures
representing curation practices such as found at ADS. Aspects relevant to the ADS
case include the emergence of practices of creolisation and
translation emerging at the boundary between
semiospheres of scholarly archaeology and data management. Other cases of
archaeological practice investigated by E-CURATORS, related to digitally-enabled
archaeological excavation and survey, may call attention to the boundary between
material-tangible and digital-intangible archaeological work, and the processes of
creolisation and translation between competing norms, methods and regimes of value
among these two semiospheres.
The E-CURATORS pilot is special in being part of a funded project, with an
international team of senior collaborators and a dedicated team of paid research
assistants, hence some of its characteristics are not transferable to the mostly
unfunded, in-kind contribution model under which DiMPO operates. From E-CURATORS, our
DiMPO protocol inherits the appreciation of, firstly, the methodological value of
zooming-in/zooming-out between different levels of DH research
activity, of attention to interactions between different actors in the context of a
situated research process - which may be identified as a research site, an
organisational unit, or a project - and between research actors and digital
resources, tools and methods. Secondly, of attention to the issues, positions and
arguments digital humanists identify as they consider challenges and opportunities,
and obstacles and drivers to purposeful DH research activity, as well as the
importance of norms, motivations and shared orientations within different kinds of
“thought collectives,” especially as DH changes through
interdisciplinarity and osmosis with pervasive digital infrastructures, diverse
stakeholders, and bottom-up practices “in the wild”
[Dallas 2016]. And, thirdly, of the shifts introduced by digital tools,
services and infrastructures to the spatial enframing of scholarly work, when
activities, methods, thought collectives, norms and other elements of scholarly
practice collectively form a conceptual rather than physical landscape, a topological
structure amenable to traversal and navigation as researchers cross disciplinary and
methodological boundaries.
Pilot Study 3: Research Community Profiles
This pilot study features our engagement with “New Exploratory
Phrase in Research on East European Cultures of Dissent”
(NEP4DISSENT),[4] a research network of
more than two hundred researchers from thirty-seven European countries, funded by the
COST Association as a 4-year Action. The Action includes a diverse group of scholars,
including historians and art historians, anthropologists and scholars of media
studies, librarians/archivists and curators, information technologists and digital
humanists.[5] One of the main goals of
NEP4DISSENT is to introduce DH methods to scholars who are already immersed in
digital materials and tools but not yet self-consciously carrying out DH projects.
Interviewees were selected who seemed “DH-curious” and were open to learning new
methodologies, but for the most part not yet fluent in DH terminology and major
approaches. Therefore, the group interviews conducted at this early stage in the
project (March 2018) served several purposes at once: (a) to measure the degree of
familiarity with DH tools within the research group, as well as probing attitudes and
beliefs about digital tools for research; (b) to provide a snapshot of scholars’
current practices and methods for collecting and curating their data (in some cases
born-digital but in most cases analogue); and, (c) to identify the key needs of this
group, in terms of areas for possible DH collaboration and training in DH methods
within the framework of the Action.
The interview protocol was developed by DiMPO members who are also taking part in
NEP4DISSENT under Working Group 5 (“Mediating Research Through
Technology”), and based on the paradigm developed earlier for the SPARKLE
project (discussed in section two above). The eleven selected interviewees, seven
male and four female, represented five disciplines: history (6), art history (2),
media studies (1), sociology (1), and tourism studies (1). They were interviewed in
groups of two to four, over videoconference, with at least two interviewers taking
part in each session. At times the subjects interacted with each other during the
interviews, but for the most part, the dialogue was one-to-one between interviewer
and interviewee. The interviews were coded in vivo by the interviewers and later
distilled into a table reflecting key aspects of the study: sources, methods, tools,
privacy, publication, barriers, and examples.
The question of barriers to digital research revolved largely around issues of access
(digitization priorities of national archives and libraries) and funding, with one
respondent pointing to a cultural issue in the region which scholars are sometimes
averse to: open-access. Another issue that came up repeatedly was the difficulty of
creating datasets that could be easily integrated with those maintained by other
scholars – sometimes because of an investment in a single platform from which it is
not easy to extract data (or at least not obvious to the researchers using it how to
do so). This does not necessarily point to any inherent problem with the platform, as
much as to the researcher’s ability to leverage its use and value. “Working with quantitative data is very time-consuming,” a
historian commented, “even though there are a lot of tools out of
there, but the risk is big because you need to build up your entire data set to
accommodate to one tool, and then you realize it’s not the tool you needed (after
you cut and pasted for days)” (Group Interview 3, 2018). These group
interviews also revealed a promising line of engagement for NEP4DISSENT to introduce
more advanced text analytical methods to those already working with quantitative
analysis of written materials, but unsure of how to engage with semantic-level
analysis. The main reasons scholars gave for using analogue over digital methods were
that: (a) their sources were too mixed, (b) that they lacked faith in digital tools,
and, (c) the scale of material collected was so small that they didn’t feel they need
for the specific tools discussed or mentioned by colleagues.
A more systematic analysis was performed by applying full qualitative data coding of
one of the group interviews. In the context of QDA, coding is defined as a systematic
classification process that helps identify themes or patterns, thus forming a basis
for the subjective interpretation of a document’s content [Hsieh and Shannon 2005].
There are different software options that enable computer-supported content analysis
and coding, including MaxQDA, NVivo, Dedoose, and ATLAS.ti. These tools allow
investigators to import various kinds of documents, usually in .txt .rtf .doc/x .pdf
formats, and to perform different actions that help the content analysis process.
For this in-depth qualitative analysis, data was gathered from the group interview
with two researchers working in the area of memory studies, both members of the
NEP4Dissent Working Group 4 on “Cultural memory of dissent” (Group Interview 4).
A full transcription of the interview was prepared and QDA was applied to the
transcribed text by using the MAXQDA software. The creation of a code system followed
an inductive (data-driven) approach by reading and summarizing raw data. The
inductive approach uses open coding, where specific codes are derived from data and
are subsequently classified in order to create the code system, usually by
categorisation aiming to reduce the number of codes by grouping them. An advantage of
the inductive approach is that it does not impose any preconceived categories or
theoretical perspectives to the ongoing research and allows to gather direct
information from study participants, which is grounded in the actual data [Hsieh and Shannon 2005]. No in vivo coding was applied during the QDA, meaning that
the process of assigning a code mainly involved paraphrasing and generalization of
selected text segments rather than verbatim codes. The inductive approach provided
useful insights into an analysis based on raw data and helped to identify potential
areas of concern expressed by researchers during the interview. Open coding resulted
in the emergence of a code system which consisted of six broader definitions (codes)
and their subcodes thus presenting the main findings of the interview (see Figure 2).
Figure 2.
Code system for the group interview analysis on digital research practices. The
inductive codes indicate five main categories related to digital research as
mentioned by the researchers in the interview, while subcodes indicate specific
details drawn from the discussion. The findings of the case study showed that the evaluation of one’s familiarity with
digital research or the role of the digital was mainly shaped by scholars’ attitudes
and beliefs, as well as by personal experience working with digital resources and
digital tools: “(...) in my case there is a disconnection extent
to which I work with digital sources and huge amount of ignorance that I possess
about them. So I don’t have proper tools and knowledge on how to work with them.
As a cultur[al] historian I see a big need to work with digital sources”
(Group interview 4, 2018).
Questions about digital resources revealed that it’s not only digital publications
that are important for research, but even more so online research data, with
different websites and social media platforms playing a key role. The Danish
researcher admits that social media is a huge incentive for her in finding new ways
of studying memory: “I also haven’t studied the memory of dissent
as such, but I was looking into dissenting memories and what you can do with
digital sources and social networks, mainly Facebook, and also Youtube. They are
places where anyone can contribute and it’s a massive resource for the study of
memory” (Group interview 4, 2018). Meanwhile, the Bulgarian scholar points
to the importance of thematic portals for her research, e. g. “(...) websites for sharing memories of socialism (...) is another source which is
born-digital, because people go there and they write their comments and memories,
but sometimes they upload photos of objects from that period (...)” (Group
interview 4, 2018).
With regard to questions of digital research methods, scholarly research processes
were the most extensively discussed by both scholars, and particular codes were
assigned to identify specific activities being mentioned, e. g. seeking of
information, acquiring and organizing research data, publishing research results,
collaborating with other scholars, and preserving data. In some cases, it’s
fascinating to acknowledge how the research process might be ordinary and
unformalized: “(..) we are very honest web users as most other
people, the way we engage with net is unprofessional acting like ordinary people
with tools that are easy available” (Group interview 4, 2018). Actually,
the simplicity of research activities was observed in every step of research, where
research data is usually acquired by copying, pasting or by printscreens and
organised by using USB stick, folder system, Powerpoint or Word document: “(...) I have a Word file where I copy-paste links that I feel I want
to go back [to]. It’s like [a] bibliography, but it’s a linkography - list of
links” (Group interview 4, 2018).
Finally, the key issues identified in the analysis of the group interview were
related to open access, lack of digital methods literacy and the complexity of online
data. And while the first two perhaps are easy to anticipate, the third issue points
to an increasing need to use online data in/for research, as well as methods and
tools that could deal with it. As noticed by one of the researchers in the group
interview, “(...) working with social networks is incredibly
tempting and definitely dangerous, because it’s so easy and the feeling is
quantifiable (you can count number of likes, comments). What I saw as a challenge
in this connection is a super-fluidiness of so much of it, it’s so unstable, so
easy, inexpensive, so you don’t know much what is going on” (Group
interview 4, 2018).
This method was also tested in an environment of scholars with a fairly long record
in DH, who identified with that field, through an interview conducted with four
members of the Stanford Literary Lab,[6] a group representing the field of
literary studies, and consisting of scholars on different stages of their careers,
with a long history of collaboration. Hence, the interview collected individual
responses but also became a platform for negotiating a collective approach towards
methods, tools and research strategies adopted by the group. The NEP4DISSENT Group
Interview Protocol was slightly modified in this study, allowing for questions
inspired by the narrative component of the episodic interviews, discussed in the
first pilot study, namely about how the researchers were introduced to digital
methods, and how they see the field in the time-perspective (five years ago and five
years from now). It should be noted that, as in the case of previous interviews, it
was clear for interviewees that the application of digital tools and methods should
be always driven by the humanities’ research questions, not the other way around.
Moreover, the group tend to seek novel research problems to develop new tools, rather
than to use the ones they have already developed on other material. This high
methodological awareness also translated into the interviewees’ self-perception as DH
scholars. Actually, in their words, being a DH scholar means having “two jobs,”
or writing “one and a half dissertation(s),” as they need to master two fields:
literary studies and computational analysis. They also pointed out to the
collaboration model of DH inquiry, which differed largely from individualistic
approaches of literary studies. They were also aware of some methodological criticism
of DH methods, which they dubbed ‘the rolling backlash’, but claimed it was very
productive for their work as it pushed them to rethink certain aspects of their
mode.
The working space is an important factor for the project workflow. Although the Lab
members often live in different cities and communicate with each other via electronic
channels, the actual meeting space remains crucial for debating the current status of
the project and exchanging the ideas that may emerge in the “first phase”
dedicated to individual work. The big table in the lab, around which the team gathers
for the project meeting, is, in the words of the founding director of the Lab, “as essential a tool as the really expensive ones ... All researchers
bring to this phase their interests, and even fixations. At times, there is a lot
of noise. But in a few magic moments, the group becomes truly more than the sum of
its parts; it ‘sees’ things that no single pair of eyes could have”
[Moretti 2016, 2–3]. Hence, the physical space becomes a platform
for the integration of the individual perspectives and generation of genuinely
collective outputs.
To sum up, this short overview of findings, interviewing DH communities may provide
more knowledge about the DH field as such, as well as about overcoming methodological
obstacles through collaboration with peers and negotiating DH status in the
humanities departments.
Pilot studies on research community profiles contributed to the protocol with regard
to establishing the research framework, desk research on actors, conducting group
interviews with researchers, as well as transcribing them, coding and analysing.The
resulting interpretations will not be designed for DH research as a whole, or even
digital history, but rather customized to fit the needs, skills, research
environment, and general methodological approaches of our project participants.
The DiMPO Case-Study Protocol
The outcome of the pilot studies is the DiMPO Case-Study Protocol for Meta-Research
in Digital Humanities (Figure 2) and accompanying guidebook (under preparation).
These materials are intended to help researchers in conducting meta-research and
adopting these guidelines for different purposes, disciplines and approaches. As many
of the issues raised here are already covered in manuals for social research, our
focus is the specificity of meta-research in the DH environment. This section
discusses only the main assumption of the protocol. For a more detailed overview
consult the guidebook (under preparation; see Next steps).
Figure 3.
The DiMPO Case-Studies Protocol for meta research in the DHBackground
The protocol should serve to bridge knowledge gaps about questions or issues that
are relevant, consequential and potentially impactful. The very first step is to
define the main area of the case study. As digital humanities tends to be
conceived as a “big tent” for a plethora of approaches
and disciplines (cf. Svensson 2012), there are
many potential areas for a meaningful work into contemporary research practices,
be it the uptake of digital encoding by scholarly editors in literary studies,
cultural analytics for art history, or databases in religious studies. The main
assumption behind scholarly practices meta-research is that a meaningful change in
scholarly practices takes place and is worth investigating. Moreover, as we argued
in the introduction, the full assessment of this change should involve not only
pioneering computational research, but rather the broader uptake of the digital
practices and methods by researchers in the humanities, even if they do not
self-identify as digital humanists, as long as they use digital resources, tools
and methods in order to explore particular problems. Hence, the investigator
should choose the most promising aspect of the field, basing on her own expertise
and knowledge.
That leads us to the identification of appropriate research questions. Sometimes,
research questions may take the form of well-defined hypotheses, to be confirmed
or refuted by operationalizing them as relationships among properties of the data
collected in the study (deduction). Alternatively, they may merely consist in
identifying relevant aspects of phenomena manifested in the data, which will be
later elucidated through “inference to the best explanation” among
alternatives (abduction). Finally, there are cases where the research questions
are little more than informal ideas and “sensitizing concepts” on potential
topics of interest, in the expectation that new knowledge will arise from the
grounded, bottom-up examination of the data (induction). DH meta-research
questions are diverse: for example, they may be how digital resources and tools
change the way research questions are formulated and scholarly investigations are
conducted, what is the status digital sources in humanities research, or what are
the effects of digital collaboration practices and dissemination of knowledge.
On the basis of prior literature and pilot studies, DiMPO identified a number of
research questions relevant for DH meta-research, categorized within the following
facets: Research Activity, Resource Assets, Tools, Researchers, Place and Time.
Facets and research questions are summarised in Table 1.
| Facets | Questions |
| Research activity |
|
| Research assets |
|
| Tools |
|
| Researchers |
|
| Place and time |
|
Table 1.
Facets and questions for the qualitative meta-research in the Digital
HumanitiesAnother integral part of this early, conceptual stage is the choice of the unit of
inquiry. DiMPO protocol distinguishes between three kinds of units, depending on
the types of research questions: research community, research project, individual
researchers. The focus on Research Community enables a view
on a particular field, discipline or community of practice, and an assessment of
the role the digital in this field. It could also assess the popularity of certain
tools and approaches in the community as well as its value judgments concerning
the evaluation of the digital work, i.e. what is considered a good practice in the
field. The Research Project/Site approach allows for a
comprehensive and formal description of a particular project, research site, or
bounded activity of an organizational unit in its entirety, in order to provide a
situated picture and understanding of a particular research endeavour, and the
interaction between people, technology and its outputs. The analysis of Individual Researchers supplements these approaches with the
biographical perspective on a researcher, her individual motivations and goals,
particular methods, beliefs held about the digital, and career trajectory. The
pilot studies presented earlier are good examples of types of differences in
results between particular approaches.
Project framework establishment
The second step is to establish the framework of the project, i.e. translate
research questions into a concrete study workflow. The questions which emerged in
the previous step will be now operationalised and particular methods and
techniques should be chosen. In the pilots we focused on personal interviews,
group interviews and observations, described in more detail in the fieldwork
section below. Once the choice is made, a researcher needs to generate the study
form, i.e. questionnaires, or observation protocols, relevant to research
questions. Additionally an ethics review should be performed and suitable policies
applied, especially in cases where there is risk and potential for harm
(professional, reputational, etc.) for study participants or investigators. Some
funding agencies may require rigorous ethics review, yet, for most of the cases,
it involves considering such aspects in clear terms as regards risks, conflicts of
interest, preservation of anonymity of study participants, right for withdrawal,
obligations for data retention, conditions of access to research data, and
consultation with study participants before publication. For instance, one should
know beforehand how raw data will be stored, or if it is going to be published in
any form. All such cases will be outlined in a consent form prepared for the
study, which should be signed by subjects prior to the interview.
Depending on the particular unit of analysis chosen for the study, investigators
will need to establish criteria for selection of cases, i.e. which particular
individuals, research projects or communities will be approached in this study.
They should consider carefully what kind of sampling technique will be relevant to
gather heterogeneous material and different approaches.
Finally, the internal work environment should be organised, which entails a
selection of tools for data collection, storage and processing, that includes
means of recording, transcribing and annotating the communication with the actors,
as well as a clear data management plan. The entire framework should be put into a
single case-study manual, which can serve as a point of reference throughout the
study for the principal investigator and the research team.
Once all of the above items are in place, a short pilot study should be conducted
to assess the research design and materials gathered in the case-study manual.
Relevant amendments should be made to the manual.
Desk research on the units of inquiry
The first phase of the actual research consists of gathering data about the unit
of inquiry (individual, project, community), which will inform the interaction
with actors and allow for the better understanding of their work. Desk research
should include a basic case overview, like research scope (what is the field),
objectives (what are the aims of research) and contribution (what are the
achievements). This includes gathering relevant materials, such as publications,
project descriptions, earlier interviews, developed software, etc. The outcome
should be presented in a form of short bios of the actors, including descriptions
of their projects and contributions, together with a bibliography of publication.
Such a resource could be later used by other team members. Materials created in
this stage could be later reused in the final report.
Fieldwork
The next stage consists of the actual fieldwork. Its scope depends on the types of
methods chosen and the size of the team conducting the study. One of the easiest
ways of acquiring material, often used in our pilot studies, is an interview by
videoconference, which is an easy and inexpensive way to reach out to a wider,
international base of subjects. But fieldwork may also include participant
observation of particular research activities, which could entail attending
project meetings and presentations, testing tools developed by actors or observing
them at work. This also can entail collection of documentary evidence, such as
photos of resources and workplaces, as well as gathering outputs of DH research
activity. Such an approach allows for gathering knowledge about the processes
which could not be noticed by subjects and not reported in interviews.
Data preparation and transcription
Fieldwork produces a diversity of raw data, which needs to be put in shape in
preparation for analysis. Field notes need to be organized and annotated with the
time and place where they were taken; recordings, photos and videos need to be
inventoried and may also need to be processed before transcription. Given the
scope of DiMPO research, audio (and optionally video) recordings should be
transcribed, so as to capture fully and faithfully the verbal content of
participant speech, but also salient paralinguistic aspects of speech, as well as
relevant non-verbal actions and events. Finally, it includes conventions for
describing gestures, pointing at objects and screen events, as these can be
important features of observed scholarly practice. CAQDAS packages include useful
transcription functions as a matter of course, allowing for slower playback and
rewind, pausing, and insertion of timestamps. Transcribers should allow for 6-7
hours of transcription time for each hour of audio recording, or roughly ten
person days to transcribe a typical set of audio recordings for each case study.
Given the multiple parameters and dimensions of DH research practices to be
investigated, the DiMPO protocol stipulates full transcription of audio
recordings, rather than open (live) transcription of only the segments that
investigators find of interest at first hand.
If anonymization has been decided on as part of the ethics review, visual material
may have to be redacted, and names of individuals in text documents and
transcripts replaced either with participant codes or, preferably, with pseudonyms
(nicknames). But, on the whole, the DiMPO protocol recommends retaining
participant identity as an important part not just of ensuring interpretability of
results, but also as an ethical commitment to giving due credit to individuals,
organizations and projects participating in the study for their frequently
innovative practices, methods and ideas.
Analysis and interpretation
The DiMPO protocol recommends that all data collected through fieldwork, as well
as in earlier bibliographic and desk research, are imported for analysis into a
CAQDAS package such as MAXQDA, Atlas Ti, nVivo or Dedoose, or a free alternative.
Segments of transcripts, documents and publications, as well as of visual records
collected for the study, can be then annotated systematically and consistently by
the use of short jottings and longer analytic memos, i.e., free text annotations
capturing the insights, questions and ideas of investigators, and codes. In the
context of QDA, a code is defined as “a word or short phrase
that symbolically assigns a summative, salient, essence capturing, and/or
evocative attribute for a portion of language-based or visual data”
[Saldaña 2013, 3]. As soon as research data from preliminary
research and fieldwork are ready for analysis, the protocol recommends that
investigators immerse themselves in the data, and produce memos which capture
initial questions, ideas and insights, as well as associations with sensitizing
concepts from the scholarly literature they consider relevant.
During the first cycle of analysis, the DiMPO protocol advocates a combination of
deductive (provisional) and inductive (open) coding. A hierarchically-organised
provisional code scheme, including ca. 50 initial codes representing the
conceptual structure of scholarly practice, will be used to encode transcript and
other data segments referring to the compositional structure of DH research
activities, their input and output, the roles and involvement of researchers and
other stakeholders, the epistemic context of their action, their motives and
goals, as well as the digital methods, procedures and tools involved in each
activity. In tandem, an inductive coding approach, based on open coding of
relevant segments of the qualitative data, can be applied to capture unforeseen
aspects of DH work, as well as issues, ideas and arguments introduced by study
participants [Miles et al. 2014]
[Saldaña 2013].
The second cycle of analysis is based on theoretical coding, and is performed
simultaneously with sorting, filtering, summarization and visualization of data
coded according to the inductive/deductive approach described in the previous
paragraph. Theoretical codes, i.e., codes that may represent possible explanatory
and causal mechanisms, underlying factors and characterizations underlying DH
scholarly practice, will be defined through identifying associations,
relationships, axial differentiations or shared categorizations of provisional and
open codes developed so far, and of the aspects of evidence represented by data
segments they are connected with. Intra-case study theoretical coding will be used
firstly to make sense of individual cases. This will be followed by theoretical
coding across cases, aimed to capture diversity and complementarity between the
practices of different DH sites or projects, rather than to achieve confirmation
through repetition. Additional fieldwork, e.g., in the form of follow-up
interviews to clarify or enrich understanding about some phenomenon that became
visible at the stage of theory building, may be planned at this point for the
purpose of saturation.
Outputs
There are three main outputs of the study: report, data, and additional
publications. The genre of a report gives more space for the inclusion of larger
chunks of evidence (esp. quotes from interviews) but should also be supplemented
by a bibliography of the publications (prepared through desk research) and a
glossary of key project concepts, i.e. its theoretical, methodological and
technological terms. Ideally, the report should contain the research protocol and
codebook, material created through desk research, discussion of the data collected
and the interpretation. A report in the field of meta-research should tackle the
questions of the role of digital resources, methods, tools, as well as services
and infrastructures in the analysed case. Reports could be stored in the DiMPO
Community on Zenodo,[7] or, if applicable, in
the DARIAH HAL collection[8] for better visibility.
Additionally, data from the project (esp. annotated transcripts) should be made
public whenever it is possible, as it could enable comparative perspectives or
reexamination of the material by other investigators (Directory of Open Access
Repositories[9] lists academic repositories
around the world).
Evaluation of the research protocol
The last step of the study is the evaluation of the research protocol, i.e.
assessment of the advantages and disadvantages of the chosen research framework
and methodological choices. This knowledge is extremely important, as it informs
further studies in the field, and — as such — should be shared in the report.
Next Steps
The main goal of this work is to enable comparable meta-research on digital
humanities, covering various disciplines. Such a body of knowledge would help in
developing a self conscious — if not self critical — approach to the research
conducted in the field. Moreover, it could serve as a methodological guideline for
non-DH researchers willing to adopt certain methods in their disciplinary work.
The DiMPO team is now elaborating a detailed online handbook to accompany the
protocol.[10] Future plans include a
series of workshops, presentations and tutorials for scholars wishing to conduct such
research. Furthermore, DiMPO will be conducting a series of case-studies deploying
this protocol. The authors are also considering whether to open the protocol to other
topics and uses within the DH community, including such specific fields as scholarly
writing practices in DH, or quality assessment. Another avenue for development would
be to adapt parts of the protocol for evaluating DH projects and generating detailed
reports about their activities, and/or for sharpening the specification of DH
development projects.
While local conditions are always the most relevant, scholarly practice increasingly
includes conceptual, rather than physical, connectivity because of norms and
competencies that are shared across geographic and disciplinary boundaries. These
collaborative practices are further enabled by online, mainstreamed and broadly
available pervasive digital infrastructures and tools (cf. Dallas 2016), which in turn produce new regimes for inter- and
cross-disciplinary scholarly work. The only approach that could conceivably handle
this variability must take into account research norms, environments (both physical
and digital), tools, social formations, and technological limitations which set the
parameters — and shape the container — in which each instance of digital research
finally takes place. Therefore, we see this paper as a step towards establishing a
systematic reflection on the dynamic field of digital humanities, a set of practices
which is more often “defined” and “debated” than studied, and which always
flows to fit the shape of the container in which it is placed.
Acknowledgements
This work was conducted within the framework of Working Group 5 of the COST Action
CA16213 New Exploratory Phase in Research on East European
Cultures of Dissent (NEP4DISSENT), and the Digital Methods and Practices
Working Group (DiMPO) of DARIAH-EU, the Digital Research Infrastructure in the Arts
and Humanities in Europe. It was also supported by the project Digital Humanities work in focus: multiple case studies of research projects
across Europe, funded by DARIAH under the funding scheme for Working Group
Activities 2017/2018.
Maciej Maryl’s research at Stanford University was supported by Fulbright Senior
Award granted by the Polish-American Fulbright Commission.
Costis Dallas gratefully acknowledges support by (a) the E-CURATORS Insight Grant of
the Social Sciences and Humanities Research Council of Canada, (b) a senior research
fellowship of the Polish Institute of Advanced Studies, Polish Academy of Sciences,
and (c) a short-term visiting research fellowship at the Long Room Hub, Trinity
College Dublin.
Notes
[2] These video interviews can be viewed here: https://www.youtube.com/channel/UCSbLhKeZXZP8Jvl2D7OxSVA
[5] For a further breakdown, see http://nep4dissent.eu/working-groups/
[10] An early draft is already available here: http://entopia.org/meta-research/doku.php.
Works Cited
Allen et al. 2011 Allen, D., Karanasios, S., Slavova,
M., 2011. “Working with activity theory: context, technology, and
information behavior”. Journal of the American Society
for Information Science and Technology 62, 776–788.
Antonijević 2015 Antonijević, S., 2015. Amongst digital humanists: an ethnographic study of digital
knowledge production.
Benardou et al. 2010 Benardou, A., Constantopoulos,
P., Dallas, C., Gavrilis, D. 2010. “A Conceptual Model for
Scholarly Research Activity,” In: Unsworth, J., Rosenbaum, H., Fisher, K.
E. (Eds.), IConference 2010: The Fifth Annual
IConference, Urbana-Champaign, IL: University of Illinois, 26–32. http://nora.lis.uiuc.edu/images/iConferences/2010papers_Allen-Ortiz.pdf.
Benardou et al. 2013 Benardou, A., Constantopoulos,
P., Dallas, C., 2013. “An approach to analyzing working practices
of research communities in the humanities”. International Journal of Humanities and Arts Computing 7, 105–127. https://doi.org/10.3366/ijhac.2013.0084
Borek et al. 2016 Borek, L., Dombrowski, Q., Perkins,
J., Schöch, C., 2016. “TaDiRAH: a case study in pragmatic
classification”. Digital Humanities Quarterly
10.
Borgman 2007 Borgman, C.L., 2007. Scholarship in the digital age: information, infrastructure, and the
Internet. MIT Press, Cambridge, MA; London.
Bourdieu and Wacquant 1992 Bourdieu, P., Wacquant,
L.J.D., 1992. An invitation to reflexive sociology,
Polity Press, New York.
Case and Given 2016 Case, D.O., Given, L.M., 2016.
Looking for information: a survey of research on information
seeking, needs, and behavior. Emerald, Bingley, UK.
Conklin and Begeman 1987 Conklin, J., Begeman, M.L.,
1987. “gIBIS: a hypertext tool for team design
deliberation”, in: Proceeding of the ACM Conference on
Hypertext - HYPERTEXT ’87. Presented at the Proceeding of the ACM
conference, ACM Press, Chapel Hill, North Carolina, United States, pp. 247–251. https://doi.org/10.1145/317426.317444
Dallas 2007 Dallas, C., 2007. “An
agency-oriented approach to digital curation theory and practice”, in:
Trant, J., Bearman, D. (Eds.), The International Cultural
Heritage Informatics Meeting Proceedings. Presented at the ICHIM07:
International Cultural Heritage Informatics Meeting, Archives & Museum
Informatics, Toronto.
Dallas 2015 Dallas, C., 2015. “Curating archaeological knowledge in the digital continuum: from practice to
infrastructure”. Open Archaeology 1, 176–207.
https://doi.org/10.1515/opar-2015-0011
Dallas 2016 Dallas, C., 2016. “Digital curation beyond the ‘wild frontier’: a pragmatic approach”.
Arch Sci 16, 421–457. https://doi.org/10.1007/s10502-015-9252-6
Dallas et al. 2017 Dallas, C., Chatzidiakou, N.,
Benardou, A., Bender, M., Berra, A., Clivaz, C., Cunningham, J., Dabek, M., Garrido,
P., Gonzalez-Blanco, E., Hadalin, J., Hughes, L., Immenhauser, B., Joly, A.,
Kelpšienė, I., Kozak, M., Kuzman, K., Lukin, M., Marinski, I., Maryl, M., Owain, R.,
Papaki, E., Schneider, G., Scholger, W., Schreibman, S., Schubert, Z., Tasovac, T.,
Thaller, M., Wciślik, P., Werla, M., Zebec, T., 2017. “European
survey on scholarly practices and digital needs in the arts and humanities -
Highlights Report”. Zenodo. https://doi.org/10.5281/zenodo.260101
Dalton and Charnigo 2004 Dalton, M.S., Charnigo, L.,
2004. “Historians and their information sources”. College & Research Libraries 65, 400–425. https://doi.org/10.5860/crl.65.5.400
Delgadillo and Lynch 1999 Delgadillo, R., Lynch,
B.P., 1999. “Future historians: their quest for
information”. College and Research Libraries
60, 245–260.
Duff and Johnson 2002 Duff, W.M., Johnson, C.A., 2002.
“Accidentally found on purpose: Information-seeking behavior
of historians in archives”. The Library
Quarterly 72, 472–496.
Edmond 2018 Edmond, J., 2018. How Scholars Read Now:
When the Signal Is the Noise. Digital Humanities
Quarterly 012.
Edmond et al. 2016 Edmond, J., Bagalkot, N., O
’Connor, A., 2016. Toward a Deeper Understanding of the
Scientific Method of the Humanist.
Ellis 1993 Ellis, D., 1993. “Modeling the information-seeking patterns of academic researchers: A grounded
theory approach”. The Library Quarterly 63,
469–486.
Engeström 1999 Engeström, Y. 1999. “Activity Theory and Individual and Social Transformation,”
In: Engeström, Y., Miettinen, R., and Punamäki-Gitai. R.-L. (Eds.), Perspectives on Activity Theory, Cambridge; New York:
Cambridge University Press, 19–37.
Engeström 2000 Engeström, Y., 2000. “Activity theory as a framework for analyzing and redesigning
work”. Ergonomics 43, 960–974.
Enserink 2018 Enserink, M., 2018. “Research on research”. Science
361, 1178–1179. https://doi.org/10.1126/science.361.6408.1178
Farge 2013 Farge, A., 2013. The
allure of the archives. Yale University Press, New Haven.
Filiciak et al. 2013 Filiciak, M., Mazurek, P.,
Growiec, K., 2013. Korzystanie z mediów a podziały społeczne.
Kompetencje medialne Polakóww ujęciu relacyjnym. Centrum Cyfrowe Projekt:
Polska.
Fleck 1979 Fleck, L., 1979. Genesis and development of a scientific fact, Repr. 11. Aufl. ed, Sociology of
Science. University of Chicago Press, Chicago.
Flick 1997 Flick, U., 1997. The
Episodic Interview: Small Scale Narratives as an Approach to Relevant
Experiences. LSE Methodology Institute, London.
Garnett and Papaki 2019 Garnett, V., Papaki, E. 2019.
Barriers and Pathways to Community Engagement (Version 1.0). Zenodo. http://doi.org/10.5281/zenodo.3956414
Grindley 2006a Grindley, N., 2006. “What’s in the Art Historian’s Toolkit? A Methods Network Working
Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp01.pdf.
Grindley 2006b Grindley, N., 2006. “Digital Tools and Methods for Historical Research: A Methods Network
Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp04.pdf.
Grindley 2007a Grindley, N., 2007. “Digital Tools for Archaeology: A Methods Network Working
Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp06.pdf.
Grindley 2007b Grindley, N., 2007. “Digital Tools for Museums and Cultural Heritage: A Methods Network
Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp09.pdf.
Hodder 1997 Hodder, I., 1997. “‘Always momentary, fluid and flexible’: towards a reflexive excavation
methodology”. Antiquity 71, 691–700.
Hsieh and Shannon 2005 Hsieh, H.-F., Shannon, S.E.,
2005. “Three Approaches to Qualitative Content Analysis”.
Qualitative Health Research 15, 1277–1288. https://doi.org/10.1177/1049732305276687
Hughes 2008 Hughes, L.M., 2008. “Conclusion: virtual representations of the past - new research methods, tools and
communities of practice”, in: Greengrass, M., Hughes, L.M. (Eds.), The Virtual Representation of the Past. Ashgate Publishing,
Ltd., London.
Hughes et al. 2016 Hughes, L.M., Constantopoulos, P.,
Dallas, C., 2016. “Digital methods in the humanities:
understanding and describing their use across the disciplines”, in:
Schreibman, S., Siemens, R., Unsworth, J.M. (Eds.), A New
Companion to Digital Humanities. Wiley-Blackwell, pp. 150–170.
Ioannidis et al. 2015 Ioannidis, J.P.A., Fanelli,
D., Dunne, D.D., Goodman, S.N., 2015. “Meta-research: evaluation
and improvement of research methods and practices”. PLOS Biology 13, e1002264. https://doi.org/10.1371/journal.pbio.1002264
Julien and Duggan 2000 Julien, H., Duggan, L.J., 2000.
“A longitudinal analysis of the information needs and uses
literature”. Library and Information Science
Research 22, 291–309.
Kaptelinin and Nardi 2006 Kaptelinin, V., Nardi,
B.A., 2006. “Acting with technology: activity theory and
interaction design, 1”. MIT Press paperback. Acting
with technology. MIT Press, Cambridge, Mass. London.
Knorr-Cetina 1999 Knorr-Cetina, K., 1999. Epistemic cultures: How the sciences make knowledge. Harvard
University Press.
Knorr-Cetina 2001 Knorr-Cetina, K., 2001. “Objectual practice”, in: Schatzki, T.R., Knorr-Cetina, K.,
von Savigny, E. (Eds.), The Practice Turn in Contemporary
Theory. Routledge, London, pp. 184–197.
Laužikas et al. 2018 Laužikas, R., Dallas, C.,
Thomas, S., Kelpšienė, I., Huvila, I., Luengo, P., Nobre, H., Toumpouri, M.,
Vaitkevičius, V., 2018. “Archaeological knowledge production and
global communities: boundaries and structure of the field”. Open Archaeology 4, 350–364. https://doi.org/10.1515/opar-2018-0022
Le Boeuf 2015 Le Boeuf, P., Doerr, M., Ore, C.E.,
Stead, S., 2015. Definition of the CIDOC Conceptual Reference
Model, Version 6.2. Produced by the ICOM/CIDOC Documentation Standards
Group, Continued by the CIDOC CRM Special Interest Group.
Leont’ev 1978 Leont’ev, A.N., 1978. Activity, consciousness, and personality. Prentice-Hall,
Englewood Cliffs, NJ.
Loesch 2013 Loesch, M.F., 2013. “Tech services on the Web: Bamboo DiRT http://dirt.projectbamboo.org”.
Technical Services Quarterly 30, 228–229.
Lotman 2005 Lotman, J., 2005. “On
the semiosphere”. Σημειωτική-Sign Systems
Studies 205–229.
Meho and Tibbo 2003 Meho, L.I., Tibbo, H.R., 2003.
“Modeling the information-seeking behavior of social
scientists: Ellis’study revisited”. Journal of the
American Society for Information Science and Technology 54,
570–587.
Miles et al. 2014 Miles, M.B., Huberman, A.M., Saldaña,
J., 2014. Qualitative data analysis: a methods
sourcebook, Third edition. SAGE Publications, Inc, Thousand Oaks,
California.
Moretti 2016 Moretti, F., 2016. “Literature, Measured, Pamphlets of the Stanford Literary Lab”. Stanford
Literary Lab.
Nicolini 2010 Nicolini, D., 2010. “Zooming In and Out: Studying Practices by Switching Theoretical
Lenses and Trailing Connections”, Organization
Studies. https://doi.org/10.1177/0170840609349875
Nicolini 2013 Nicolini, D., 2013. Practice theory, work, and organization: an introduction,
First Edition. Oxford University Press, Oxford.
Palmer et al. 2009 Palmer, C.L., Teffeau, L.C.,
Pirmann, C.M., OCLC Research, 2009. “Scholarly information
practices in the online environment: themes from the literature and implications
for library service development”. OCLC Research, Dublin, Ohio.
Perkins et al. 2014 Perkins, J., Dombrowski, Q.,
Borek, L., Schöch, C., 2014. “Building bridges to the future of a
distributed network: From DiRT categories to TaDiRAH, a methods taxonomy for
digital humanities”, in: International Conference on
Dublin Core and Metadata Applications. pp. 181–183.
Pertsas and Constantopoulos 2017 Pertsas, V.,
Constantopoulos, P., 2017. “Scholarly Ontology: modelling
scholarly practices”. Int J Digit Libr 18,
173–190. https://doi.org/10.1007/s00799-016-0169-3
Reimer 2009 Reimer, T., 2009. “Classifying the (digital) arts and humanities”, in: 2009 5th IEEE International Conference on E-Science Workshops. Presented
at the 2009 5th IEEE International Conference on E-Science Workshops, pp. 197–197.
https://doi.org/10.1109/ESCIW.2009.5407979
Saldaña 2013 Saldaña, J., 2013. The coding manual for qualitative researchers, 2nd ed. SAGE, Los
Angeles.
Schreibman et al. 2016 Schreibman, S., Siemens,
R.G., Unsworth, J., 2016. A new companion to digital
humanities.
Shum et al. 2006 Shum, S.J.B., Selvin, A.M., Sierhuis,
M., Conklin, J., Haley, C.B., Nuseibeh, B., 2006. “Hypermedia
support for argumentation-based rationale: 15 years on from gIBIS and QOC”,
in: Dutoit, A.H., McCall, R., Mistrik, I., Paech, B. (Eds.), Rationale Management in Software Engineering. Springer-Verlag/Computer
Science Editorial, Berlin, pp. 111–132.
Stake 2013 Stake, R.E., 2013. Multiple case study analysis. The Guilford Press, New York.
Stone 1982 Stone, S., 1982. “Humanities scholars: information needs and uses”. Journal of Documentation 38, 292–313.
Svensson 2012 Svensson, P., 2012. “Beyond the Big Tent”, in: Gold, M.K. (Ed.), Debates in the Digital Humanities. University of Minnesota
Press.
Unsworth 2000 Unsworth, J., 2000. “Scholarly Primitives: What methods do humanities researchers have in
common, and how might our tools reflect this?”, in: Humanities Computing: Formal Methods, Experimental Practice Symposium, King’s
College, London. Presented at the Humanities Computing: formal methods,
experimental practice symposium, King’s College, London.
Vukotic 2015 Vukotic, A., 2015. Neo4j in action. Manning Publications Co, Shelter Island, NY.
Wikipedia 2018 Wikipedia, 2018. “Meta-research”. Wikipedia.
Woolf and Silver 2017 Woolf, N.H., Silver, C., 2017.
Qualitative Analysis Using MAXQDA: The Five-Level QDATM
Method. Routledge.
URL: http://www.digitalhumanities.org/dhq/vol/14/3/000477/000477.html
Comments: dhqinfo@digitalhumanities.org
Published by: The Alliance of Digital Humanities Organizations and The Association for Computers and the Humanities
Affiliated with: Digital Scholarship in the Humanities
DHQ has been made possible in part by the National Endowment for the Humanities.
Copyright © 2005 -
Unless otherwise noted, the DHQ web site and all DHQ published content are published under a Creative Commons Attribution-NoDerivatives 4.0 International License. Individual articles may carry a more permissive license, as described in the footer for the individual article, and in the article’s metadata.
Comments: dhqinfo@digitalhumanities.org
Published by: The Alliance of Digital Humanities Organizations and The Association for Computers and the Humanities
Affiliated with: Digital Scholarship in the Humanities
DHQ has been made possible in part by the National Endowment for the Humanities.
Copyright © 2005 -
Unless otherwise noted, the DHQ web site and all DHQ published content are published under a Creative Commons Attribution-NoDerivatives 4.0 International License. Individual articles may carry a more permissive license, as described in the footer for the individual article, and in the article’s metadata.
