There were 108 responses to the survey. Of those, 63 were complete responses — in other words, all questions in the survey were completed. This represents several individuals who returned multiple times to describe different tools (in total 54 individuals completed the survey). Of the four respondents who filled up the survey multiple times, one person described six tools, two described three tools, and one described two tools.

Partially completed surveys that were partial because the respondent was returning to discuss a second or third tool, or because the respondent had skipped a question that was not relevant, were kept. Partially completed surveys in which the respondent had only filled out the consent form and demographic information were discarded.

Forty-eight respondents answered the multiple-choice question about who they were: 32 respondents (67%) identified themselves as Faculty (teaching). The next highest was Programmer/Developer with 6 respondents (13%). Libraries (faculty or non-faculty) came in third, with 4 respondents (8%). Departmental affiliations were entered by the respondent (i.e., there was no controlled vocabulary): English departments were the most represented (11 respondents); Libraries and Information Studies were next (eight respondents); six identified their primary affiliation as a Humanities Center.

Tool development was categorized in four broad areas: text analysis, authoring, teaching, and visualization. Interestingly, the percentage of tools developed in each category were remarkably similar.

Other categories that were not captured but which had more than one response were database management tools, indexing tools, and archiving tools.[2] Many of the tools developed by practitioners in the digital humanities community are represented in the survey. These include tools with “brand” names many might recognize: Hyperpo, Image Markup Tool, Ivanhoe, Collex, Justa, nora, Monk, Tact, TactWeb, Tamarind, Tapor, Taporware, teiPublisher, TokenX, Versioning Machine, and Zotero. It was clear that frequently more than one project participant filled up the survey. Equally, many respondents developed “unbranded tools,” simply describing them as scripting tools, perl routines, tei stylesheets, or exercise authoring tools.

This latter category seemed to the authors to be the more invisible side of tool development. While branded tools frequently gain public recognition through their websites, public announcements on mailing lists, conference presentations, and published articles, tools developed to do particular routines or to make work easier for a particular developer or project, are less frequently considered in discussions in the academic system of rewards particularly to those on tenure or tenure track lines.

There does appear to be some correlation between branding tools and the perception that tool development has contributed to career advancement. There were twenty-two “yes” answers to the question, “Has your tool development counted towards career advancement (i.e., it has counted towards tenure or promotion)?” Of those 22, it was possible to correlate 18 with degree of collaboration on tool development and name or description of the tool developed. Of those 18 individuals, 8 reported that they had developed tools with “brand” names. Compare these numbers to those who answered @no to the career development question — out of 11 who felt that tool development had not helped advance their careers, only 2 reported working on tools with “brand names.”

Our presumption going into the survey was that tool development in the Humanities is an inherently collaborative activity. The results bore this out: 50.8% of respondents described their collaboration as extensive. Interestingly, 15.3% described collaboration from not at all to moderately (or 1-3 on a scale of 5). In terms of whom collaboration was with, 85% were with programmers, with nearly 80% of collaboration with colleagues in the humanities. In one way this is not surprising: a majority of humanities scholars probably need to collaborate with programmers in tool development. What was unexpected was that nearly as many respondents indicated that their collaboration involved humanities scholars. As only 12% of respondents self-identified as programmers/developers, it seems logical to presume that tool development takes place within teams in which there is more than one humanities scholar.

Overall, survey respondents responded positively regarding the success of the tools they had developed. 94% of the respondents (54 of 58) said that the tool they had developed still fulfilled its original purpose. A lower percentage (though still a majority) reported that they considered their tool development endeavors successful — 33 of 54 indicated “yes” (61%); 21 said “somewhat” (39%); interestingly, not one respondent ticked the “no” box.

Respondents who felt their tools were successful described how they measured the success from a controlled list. The highest ranked answer was “ability of the tool to do the job it was intended to do” (with 96% of respondents choosing that option). The next highest option was that the “tool enabled the respondent to further their research,” (with 78.4% of respondents choosing that option). The lowest choice, but still with a majority (52%) measured success by the “number of users who have adopted the tool.” Nine respondents chose “other” and their answers tended to focus on tool development as an activity that enables further research.

Twenty-one respondents checked that their tool development was “somewhat successful.” The reasons for their lack of success fit, by and large, into one of three categories: lack of resources or funding (5 answers); too early in development to determine success (5 answers); and the tool was somehow not suitable for users — either not adopted by many users or not as useful as intended (5 answers).

While we were interested in the value developers placed on their own tools, we were also interested in how valuable they felt the development of tools was to their own research and how they perceived the value placed on these activities in terms of scholarship, promotion, and tenure. All of the respondents answered the question, “Do you consider tool development a scholarly activity?” 51 respondents, or 94%, said yes. The three respondents who answered no gave these reasons:

1. didn’t want to check “yes” or “no” — consider it more a service activity than scholarly activity;
2. one respondent focused on their particular tool, which they said was only a “response to needs for workable data management”;
3. the third respondent wrote that “it does not in itself advance scholarly knowledge; it is implicit in the word ‘tool’ that its just a tool which other use to do scholarly activity.”

More interesting, however, were the detailed responses to the question for those who answered that they derived scholarly benefit from tool development. Respondents cited benefits such as a better understanding of source materials or processes (11 of 51 or 21%); creating traditional intellectual output, such as publications and conference papers (6 of 51, or 12%); cognitive benefits — such as a better understanding of analytical methods, systematic reasoning, and “the problem space” (3 of 51, or 6%.) Several responses clustered around the area of creativity. One of the respondents answered that the tool is a community/public artwork/creative work that enables interdisciplinary collaboration, while another answered that tool development furthered the relations between objects, making it a playground for innovative user interfaces and browsers with visualisation capacity.

Two responses described deferred benefits. One simply replied, “Awaiting benefits.” Another deferred benefit was tied to traditional publication: “None so far but we will publish in journals later.” One respondent’s experience was clearly mixed when describing the scholarly benefits derived from tool development: “None,” was the reply: “Was never promoted and lost salary, but was amply rewarded by the sales of the product.”

When asked if tool development has counted toward career advancement, respondents were less certain (although more confident than our assumptions going into the survey). All of the respondents answered this question: 22 (40%) ticked “yes”; 18 (33%) responded “don’t know”; and 14 (26%) ticked “no.” We correlated those responses with the respondents’ departmental affiliations to see if patterns emerged. Of the two departments most represented in the survey, those in English responded with six “don’t know”s, three “yes”, and only two “no”. Library and Information Science was similar, with four “don’t know”s, two “yes” and two “no”. Those who identified a Humanities Center as their primary affiliation were most confident as a whole, with four “yes” and one each for “don’t know” and “no.”

There were several repeated themes among the 51 respondents who elaborated on their answers: the most-often-cited positive responses included raising one’s profile in the field; winning grant money; increasing avenues for publication and presentations; gains in salary; and new job opportunities. One respondent concluded that it must not have hurt as “I continue to be gainfully employed.” Others were more positive: “It was an important part of my tenure package and understood well in my department.”

However, at least four respondents felt that despite the other rewards they might have gained, promotion or tenure were less likely due to the inability of tenure committees to properly evaluate tool development as a scholarly activity. Other responses were even more succinct as to the negative impact the activity had on their careers: “It derailed it”; another answered, “For whatever reason, the University never rewarded this activity.”

Other responses hint at the grey area that many practitioners work in — neither knowing if their development activities hinders or helps: “I keep including tools in my professional reports, but I doubt the time, intellectual investment, and impact are considered. I keep hoping for a turning point . . .”

Another grey area emerged from a closer examination of degree of collaboration reported in the survey. The average degree of collaboration was somewhat higher among respondents who reported that tool development had advanced their careers than among those who reported that it hadn’t. On a scale of 1 to 5 (with 5 being the highest level of collaboration) the combined average for the “no” group was 3.09 while the “yes” group was 3.68. The “yes” group included 9 of 19 respondents reporting “5,” the highest level of collaboration, with only 2 reporting “1” or essentially no collaboration. In the “no” group, 4 of 11 respondents reported a “5,” and 3 reported “1” or no collaboration. What our survey cannot tell us, however (and this may be future research), is whether those who reported both a high degree of collaboration and career advancement felt that their advancement stemmed from the collaborative development of the tools themselves, or from single-author papers and presentations that resulted from their work. In other words, is collaborative, cross-disciplinary work beginning to be better recognized and rewarded in disciplines that value single-author works, or are the secondary outputs of that work — i.e., articles and presentations — being rewarded?

We were also interested to know what avenues were used to publicize tools and what percentage of tools were developed for public use. The vast majority, 84% (48 responses), replied that the tools were made available to others. When asked about obtaining feedback on the usefulness of the tool, as a whole, tool developers were less systematic. The vast majority of respondents indicated that their main mechanism for feedback was asking colleagues (86%), while 47% obtained feedback via the tool website. Less than a third of the responses (31%) indicated that usability studies were conducted, and even fewer, 14%, utilized surveys. Mailing lists, or more specifically project mailing lists, were frequently cited as a mode for obtaining feedback.

Dissemination of tools was along traditional scholarly lines, with 75% responding that they made their tools known via conference presentations. This was closely followed by the project website (71%), with word-of mouth- dissemination at conferences coming in third at 64%. Not surprisingly, Humanist was the most frequently cited mailing list, with TEI-L coming in second.