Most participants felt that the About the Author and Author’s Work tools helped them to evaluate the article they read by providing them with a basis for judging the position and contribution of the author. While a number of participants did recognize the author of the article they were reading, ten participants commented that if they did not recognize an author’s name, they would be interested in gaining more information about the author’s background and current research activities. Ten participants commented specifically on the usefulness of the About the Author tool in evaluating the article, and author, in question. Seven users also commented favorably on the Author’s Work tool. While these domain-expert readers clearly trust their own judgment of an article, they are also interested in checking those judgments against easily ascertained signs or markers of expertise in the author’s record. When using About the Author, a participant pointed out, “it can lend credibility to the article knowing who the author is, because – and for me specially this is an area that I don't know – I know the big names but I didn't know for example [this author’s] name before reading this article”  (P5). In response to the Author’s Work tool, this participant stressed the importance of the venue in which the work was published but this was not the participant’s only concern: “It’s a question of credibility and knowing what he's working on. And the fact that he's published in other respectable venues[;] that he's somebody that does interesting stuff. I mean just reading a rather long abstract told me that . . . this is a guy who is doing important work”  (P5). This telling comment reflects how a domain-expert reader arrives at a judgment by working through multiple signals, from the name-recognition of the journal to the careful assessment of an abstract from one of the author’s other pieces. Two other participants P3 and P9 commented that they were more likely to look to the abstract than the metadata for certain kinds of information, such as keywords. It is worth noting that the abstract is a rather recent phenomenon in humanities journals (given this field’s reliance on the literary tradition of the essay rather than the writing up of an experiment). This participant’s ability to conduct a relatively quick check on the overall quality of the author’s work, with the help of “a rather long abstract,” speaks to the potential value of the abstract to the creation of indexing information when the full text is not available.

Paying attention to the “respectable venues” in which authors publish proved a common theme across a number of tools that involve searching the journal literature, including the Related Studies, Databases, and other tools. Eight participants commented on publication venue as being a factor in their evaluation of the author’s article and other works. It became very apparent that where an article is published matters a good deal to these readers. Initially, a journal’s peer-review process provided domain-expert readers with a pre-review and preliminary screening. And this pre-review – that is, prior to readers coming to their own judgment about an article – becomes all the more reliable through the readers’ rating and rankings of journals within their field.[6] These readers place a great deal of emphasis on the journal’s assessment of an article when performing their own evaluation: “I evaluate quality and significance partly by where things appear, which is the major problem that online publishing has”  (P1).

The growth of online publishing has challenged many of the assumptions readers hold about “where things appear.” Peer-reviewed work is appearing in new sort of a journal (e.g., Borderlands eJournal: New Spaces in the Humanities; Computing in the Humanities Working Papers). This new realm may pose a challenge to domain-expert readers who have been trained within a print culture: “Personally, I have [a] fair amount of skepticism of what's available there on the web unless I'm quite familiar with the sites and if they're discipline specific and created by academic institution or trusted institutions”  (P12). There’s reason enough for such skepticism. This wariness only adds to the work of arriving at a judgment about the article being read, as well as to the chance that one is wasting one’s time with it.

The problem of online credibility has been compounded, particularly in the case of the humanities, by the relative slowness with which the standard journals in this field have become available on the Web. Many in the humanities today continue to believe that “online publication is still having real trouble getting accepted as... first rank”  (P1). Fortunately, journal-hosting services such as Project Muse are now providing online access to a greater proportion of the peer-reviewed literature in the humanities even as the journals also remain in print for the present time.[7] Eventually, scholars will be able to settle back into their trusted ranking schemes. In the meantime, it is clear that the reading tools need to do more to identify the standards that govern its sources, as noted by one of the participants:

What is peer-reviewed or not, we're not always sure. What is a good article or not in the field, it's very hard to define – much of it is . . . either password protected [or requires] pay. So, it's difficult to find and get to certain resources that might help me evaluate this article. Certainly, you can get to tools that might help to analyze this article, but evaluate – that's really a knowledge issue, not much of a tools issue.  (P12)

The participant’s point is well taken. The actual evaluation of the article depends not on the tools but on the reader’s knowledge and experience. This point was reinforced by yet another participant in the study, who concluded that using the reading tools “reinforced an opinion [he] already [has] that throwing automated processes at text is not very valuable. Whereas having humans read text and make judgments and decisions about them is a) useful [and] b) what we're supposed to be doing”  (P9). However, it might be noted that as the reading tools provide readers with related open-access peer-reviewed articles, the tools augment rather than hinder the reader’s ability to “[make] judgments and decisions” about the work being read. Of course, more can be done to enable readers to gather background information readily on the sources used in the reading tools. As one participant pointed out, “that's what people check, they check editorial boards to see if they recognize the names and if they seem to be people from well known places”  (P1).

The reading tools link to information about each database in order to provide readers with the means to evaluate each source. This link takes the reader to the database’s homepage where in many, but not all, cases readers can discover more about how materials have been selected to be included in the database. While the About link works to describe the database, information about individual journals must come from other sources. Many online journals, however, carry with them the authority attached to their print-based origins.

For all their apprehensions about the status of online publishing, readers expressed support for other web-based means of communication such as online forums where scholars can gather for informal exchanges and commentaries. In the humanities, online discussions through such services as H-Net have long been a vital part of the field’s use of online resources. The Online Forums tool directed readers to discussions thematically related to the article they read. Participants considered online forums to be more current and less “stilted”  (P4) than other publication venues. Regardless of the article’s publication date, the reading tools allow readers to tap into the most current discussions of the article’s themes as well as any archived conversations. Online discussions reassured readers that they are part of a community. One participant commented, “you're not alone in front of your computer and that's why I like all these sites where you can comment and you can add and so on . . . it’s very good to be part of a community on the internet”  (P6). Another participant noted that this tool provided access to a level of engaged and incisive discourse that could not be found in the journals and described the value of such timely and current discussion in the evaluative process:

There's open discussion and informal discussion on a wide variety of areas that you just won't find in some of the other academic publications . . .. It'll give you some insight on academic questions around the discipline that again you may not find in formal academic publications. So I think the Forums give you, in some ways, a more cutting edge insight into the people involved in the discipline, the things that are occupying them right now versus their book that published two years ago, three years ago, and who is within the community.  (P12)

This participant made a connection between comprehending an article and understanding a field. The reader commented that seeing “other works by the author,” as well as coming across “other people and disciplines who might be discussing works of this kind” and “discussions or arguments for this particular topic” in online discussion forums would “help you to understand your article” before concluding that “the more that you can find within this context . . . the more you will learn about the discipline and the more you are able to fit this article within that context”  (P12). The community nature of an online forum could help users understand both articles and the field by fostering a dialogue with other researchers in the field, as they might take a moment to add to the discussion as a result of their current reading.[8]

Despite the favourable comments on online discussions, opinions on the Comment tool, which enables readers to leave comments on the article they were reading, were mixed. Five participants felt that being able to read opinions from other readers would be beneficial. One participant noted that leaving a comment could “create a scholarly conversation”  (P5). Six participants, however, were skeptical as to how often they would use such a feature. As one participant noted, “I would very rarely want to bother making comments unless there's an active discussion already going on or there is likely to be an active discussion going on. I think the idea of accumulating layer after layer [of] interpretive, reactive stuff around articles is a bit, I mean it sounds cute, but how much of that stuff would anybody ever wade through in real life?”  (P9). In a similar vein, while most participants seemed willing to email the author of an article with ideas they had about the work, the built-in Email Author tool was not felt to be necessary. One user, for example, commented that he would expect to find that link at the end of the author’s biography rather than as a separate tool P12. Another participant suggested the possible use of an “add comment” feature that is configured to notify the author through email P4, revealing a familiarity with content management software that is increasingly common.

A number of participants commented on the value of the reading tools for teachers and students. One participant stated, “I can see that some of these tools might be more useful to a student, than they would be to somebody who already has a fair understanding of the field”  (P3). Another participant noted, “if I’m using [an article] for teaching . . . I like to put it in a larger context. So then [the Literary Critics tool] would help me relate it to other materials and to show how it was evaluated by other people”  (P10). Other participants praised the Literary Critics tools as a valuable resource for students looking for supplementary material. One commented, “it’s often very difficult to find good secondary references especially in a field like Humanities computing that’s growing and in essence the last body of work [made digital] is criticism”  (P12), while another felt that “being able to click on and find the other works of an author, the critic, that they’re interested in is extremely helpful [for students]. And it helps [students] realize the scholarly context in which the person is working”  (P5). Finally, a participant felt that access to dissertations would be of value to students who could use them to find out what research is currently being done in an area without waiting until the work reaches a published state P2.

Students were also seen as the major beneficiaries of the Define Terms and Dictionaries tools, which allow readers to look up individual terms in online sources: “I loved the [Define Terms] tool for student reading of academic writing and I would find that a wonderful tool, and I think I might really enjoy it for myself too in reading a more technically complex article”  (P3). There was also a relatively rare instance of a participant seeing a crossover value for the tools between domain-expert and non-expert readers. For example, a second participant made it clear that she had already worked out a way to define terms readily online, but that these reading tools would benefit students P6.

The question of scholarly quality and restricted access was raised again, pointing to expert readers’ emphasis on open access to research tools: “All you can do [with the Define Terms tool] is search Miriam [sic] Webster which is not a particularly good dictionary...if I were looking up a word, I would go to the OED [Oxford English Dictionary] online which my university has access to”  (P1). The same participant expressed similar concerns about the Dictionaries tool (which should rightly be renamed the Encyclopedia tool as was pointed out by P10), making it clear that the range of reference material made possible by library subscriptions was far superior to what was currently available through the reading tools. This shortcoming in the reading tool’s ability to take full advantage of these readers’ online resources is now in the process of being addressed.

As domain-expert readers, participants exhibited techniques for augmenting and supporting the judgments they make about what they are reading. Some were already in the habit of turning to Google and Google Scholar. One asked, “rather than bothering going through all the separate tools [if] I wanted to find out books on the same topic, why wouldn’t I just go to Google Scholar and be done with it?”  (P7). A second participant put it plainly: “[rather than use the Author’s Works tool] I would have done actually something simpler than going through all these searches . . . I would have gone through to Google Scholar or the online catalogue or library”  (P6). Finally, a third participant commented, “so it's just giving me a different way of searching”  (P2). The value of the tools for the domain-expert reader would seem particularly limited for those possessing more than a modicum of web savvy. Participants with less web experience were correspondingly less likely to compare the reading tools to Google. For them, however, the tools posed their own technical challenges. One participant commented that despite wanting “to be really positive here,” we had to recognize that introducing these tools was “just making [his] life more difficult”  (P3).

However, the participant who felt the tools just provided a different way of searching, which she knew how to do well enough, went on to pose something of a challenge for the design and reach of the tools. She pointed out that the tools did not open up new ways of reading for her; they were “a useful idea, but it doesn't change my perception of what's possible”  (P2). This prospect of changing the domain-expert readers’ sense of connections and contexts had been a hope with the tools. The change was intended in particular through the tools that connected the research article to government policies, media stories and instruction sites, and that connect readers to these themes in a focused way that is often difficult to obtain from broad Google searches. While none of these participants identified the Media Stories and Government Policy tools as particularly useful to their own research, which is understandable in most cases – “there's nothing in the government of Canada archives that's relevant to me on… Chaucer's text, right?”  (P1) – four participants saw these tools as being of potential use to researchers in other fields. At least one participant recognized the value of these tools for the very difference it introduced: “I think that the media report, or the notion of having media articles there to see what the kind of public or the press is saying about the area is useful because it gives a very different perspective”  (P2).

Nine of the participants commented on the inadequacy of the provided search terms. These terms, you may recall, are normally drawn from the first two keywords the author provides for indexing the work in the process of submitting it to the journal, but in this case they were based on our reading of the article. Participants felt the terms provided were either too general to be useful or were actually misleading. In addition, one of the distinguishing features of domain-expert readers reading in their own field is how they “automatically constructed the main idea significantly more often when reading texts about familiar topics,” which is to say that they are capable of generating very accurate keyword searches on their own [Afflerbach 1990, 43].

One participant commented on the lack of focus in the search terms: “The key words both in the search scenario and in the metadata material I found almost too broad to be of assistance with this particular article”  (P2). Another observed that viewing the metadata provided for her article “[explained] why most of the search tools later on failed as [abysmally] as they did, because it’s improperly categorized”  (P1). This participant added, “the direct link [to Google from the OJS] isn’t so useful because the search terms in his case probably wouldn’t be the ones I’d choose to use anyways”  (P1). What needs to be noted here is that readers, before clicking “search” with any of the reading tools, are able to edit, delete or add search terms. Apart from overlooking this feature of the tools, one participant did go on to suggest that we should incorporate metadata automatically generated from the words used in the article P4.[9] Another felt it was the responsibility of professional cataloguers to read each article and determine categories and key words P5, which would greatly increase the accuracy of the reading tools, without question, while adding to the expense of running a journal.

Certainly, there is more that can be done to improve the indexing of the articles, and thus the precision of the searches. The journals' editors are able to provide authors with examples of indexing terms specific to their field, and they have the option of including a link to a thesaurus (e.g., ERIC Thesaurus in education) or classification scheme (e.g., Mathematics Subject Classification) that can further guide authors. It is in the authors’ self-interest to learn more about how to accurately index their work. Another approach would be to draw on “folksonomy” tagging – used with blogs and social networking websites – in which readers can collaboratively suggest indexing terms for a work. These indexing terms can be used to create a word cloud which reflects the frequency with which each tag is used. Readers using the tools would then select tags or indexing terms from the cloud as their search terms.

One feature of reading research articles online that the project did not yet have in place when this study was conducted was found to be basic to the work done by researchers reading journal articles. Researchers check the works which the author references not only to see if the reference has been quoted and otherwise treated fairly, but because the references to an article provide one of the best sources of additional information on the topic. The reading tools operate independently of the article’s references in the sense that the reading tools find related materials that have appeared after the article was published and that are potentially from a greater variety of perspectives on a topic than the list of references that an author assembles in writing an article.

Nonetheless, one participant in this study pointed out the value of having similar sorts of links from the reference in the article to the work referenced in the context of teaching a class on the original article: “[If] you have one text and the author is comparing it to another . . . you might zip over to have a look at the other text to see whether you want to discuss that one as well”  (P14).[10]

What became very clear from our interviews with the participants is that offering them the best of what was available in an open access format for their field fell well short of what they were used to, even in online environments. As faculty members and graduate students at research universities, they were able to log into their library’s online resources from their offices and their homes, not to mention hotel rooms and airports. As a result, some participants compared the databases offered by the reading tools to those available through their library’s subscription: “There are much better tools than the ones that are included [in the reading tools]. Now, if this included the MLA, then sure it would be useful, it’s just that right now, the tools that are there aren’t the ones that would give a complete listing of his work”  (P1). One participant summed up the reading tools’ open access limits this way: “I think it’s an insurmountable problem . . . until this stuff really becomes free, because everything that’s free is incomplete . . . and I care about complete”  (P1). Dissertation Abstracts International, Oxford English Dictionary, and Project Muse were among the other subscription services that the participants felt were sorely lacking from the reading tools, given their dependence on open access databases. Another participant questioned the choice of the Internet Public Library as the only resource accessed by the Literary Critics tool and wondered why journal databases were not included P2.

However, there is a solution to this problem. To address this obvious and substantial shortcoming for domain-expert readers, the project is developing an OpenURL framework for the tools. This is what Google Scholar has done, enabling faculty members and students to register their library with Google Scholar, which then, with each search, identifies the items found that are in the library. It will take an extra step, but then so does logging into one’s library from a remote location. The result will be, to use the words of one participant, a “comprehensive list [with] an indication of which items I can get at my desktop, and which ones I actually have to go to the library for”  (P1).

At least a few of the participants showed great sensitivity to the access question. They were interested in providing a way for more than just scholars and students to be able to take advantage of these scholarly resources. That is, they were supporters of the open access movement in scholarly publishing: “What I'd hope [is] that . . . you could get academics hooked into the system so they'd publish copyright free articles under this system[;] in that [sense,] information will be free”  (P4). Another participant drew the distinction of the university’s obligation to the larger public, within the economic reality of publishing: “I'm very sensitive to the notion that publishing, even electronically, can cost money… but we're also working in a university environment where what we're doing is usually made available to the public; . . . the fact that I couldn't get this for free anywhere at all is problematic to me, philosophically speaking”  (P5). This participant also stated on using this tool that “I think most of us . . . have a preference for . . . stuff that’s open…. I don’t object to having it on here and if people do want to pay I guess they ought to be able to [but] at the same time I would hate to encourage this sort of thing”  (P5). There were also participants who responded negatively to the Pay-Per-View tool, which was the name of the only non-open access tool included among the reading tools: “I’d probably have to be pretty desperate to use [the Pay-Per-View tool] . . . even if we click on it, we don’t seem to be getting very much”  (P3).

As this was the participants’ first experience with the reading tools and there were limits to how thoroughly the use of the tools could be reasonably explained to them, there were times when what the tools could do were misunderstood. Still, these misunderstandings provide insights into domain-experts’ reading habits:

I have my own way of organizing my information on my screen and one of the things would be to bookmark something. . . . so if I lose my ability to bookmark, because I'm within this window that's not giving me those features, then I can't save for myself a lot of material that I would want to be able to come back to.  (P2)

The ability to bookmark the databases provided by the reading tools, as well as the individual articles found within any one database, could have been done at any point by using the browser’s Bookmark feature, which of course operates independently of the reading tools. Still, this participant’s interest in archiving and organizing the results of reading-tool searches does suggest that more could be done to facilitate this process. In an earlier version of the reading tools, readers could establish an online archive in which an article’s metadata could be stored, whether readers were on their own computers or not. Highwire Press includes a Personal Archive link for articles as part of, for example, the New England Journal of Medicine. The point is that domain-expert readers are always in the process of building resources as a result of their reading. These readers keep an eye on the long-term utilization of the knowledge they encounter.

What this study makes clear in relation to earlier studies of the cognitive processes that domain-expert readers apply to reading within their domain is that this reading process includes positioning and assessing the author and his or her ideas in relation to the larger field. The reader follows the flow of ideas within the article, all the while mapping and judging those ideas in relation to work with which they are already familiar and new work they have encountered through the reading tools. The judgment side of this process relies on finely tuned measures of hierarchical ranking. These measures are continuously recalibrated as the reader evaluates the author’s other work, where the author’s work is published, the author’s institutional affiliation, and other external factors. This aspect of the reading process is aided by the use of the reading tools, much as the field-mapping and evaluative footnote championed by Edward Gibbon aided readers during the Enlightenment.[11]

Within the scope of this domain-expert reading process, a number of tools, such as Author’s Work, Author Bio, and Related Studies, have proven capable of providing additional, readily accessible indicators or signals of the potential value of article they are reading. If not in every case, some readers used the tools to consult a number of sources, virtually on the fly, without losing sight, as it were, of the original article they were reading. Once having read the article, they may have come to reassess the initial judgments made on the basis of what they found through Author Bio and Author’s Work. However, constant self-reflection is what distinguishes such readers [Wyatt et al. 1993]. Still, we should not assume that acts of reading scholarly articles always coincide with making sense of disciplinary fields, even for domain-experts. One participant drew this very distinction between comprehending the article and learning more about the field: “I read an article and I understand it . . . finding out more doesn't change my understanding of the article. It may change my understanding of the field, which is a bit different”  (P1). The participant’s reflections reiterate the interest of such readers in learning about who is doing what and where. This added contextual material makes for a complex reading, learning, and evaluation process.

For the domain-expert reader, learning to read a field means attaching ideas to people and vice versa. It means doing so within an evaluative framework that critically identifies not only who is associated with whom, but also which ideas are most promising and which are passé. These readers fix ideas to people, as well as to particular works, in part because this is necessary for utilizing the ideas within their writing and teaching; they are distinguished as readers, in other words, by their adherence to rules of attribution and to notions of intellectual propriety and property. Further design work on the reading tools, as has been suggested above, will seek to improve the tools’ ability to assist domain-expert readers in learning more about the author and the work being read. The tools seek to augment, however slightly, these readers’ deep knowledge of how their field is structured. In this way, the tools are intended to extend these readers’ critical engagement with their field of expertise.[12]