Our publicity campaign targeted a variety of audiences including the general public, academic community, libraries, archives professionals, and schools. We devised audience-specific tactics as well as more general strategies, taking advantage of services offered by UCL to help us implement our campaign; these included the various Media Relations, Corporate Communications, Outreach, Public Engagement, and Learning and Media Services teams. In devising these strategies we had to consider issues of cost and timing. Transcribe Bentham had a limited budget to spend on publicity – £1,000 – and, as our testing period was six months only, a short time-frame in which to execute the plan. Though we hoped to target the English-speaking world, many of our strategies were, by necessity, confined to the United Kingdom.

During the six months as a whole, the Transcription Desk received 15,354 visits from 7,441 unique visitors, or an average of 84 visits per day (see Figure 7).[23] Period One saw 5,199 visits, while in Period Two, there were 10,155. It is clear, then, that traffic to the site during the shorter Period Two was much greater than the longer Period One, but this is far from the full story.

Following the publicity surrounding the launch of the Transcription Desk, there were 1,115 visits to the site during the first week, though things settled down soon afterwards when, during the remainder of Period One the site subsequently received an average of forty visits per day. Indeed, in November and December the number of daily visits rarely rose above thirty, on occasion reached sixty, but dropped as low as seven during mid-to-late December. Traffic to the Transcription Desk had essentially flatlined, though the volunteers then taking part had transcribed 350 manuscripts by the time Period One ended.

Then came the NYT article. From eleven visits on 26 December, traffic rocketed to 1,140 visits on 27 December, with a further 1,411 the following day. Remarkably, thirty per cent of all visits during the testing period to the Transcription Desk came between 27 December 2010 and 4 January 2011. The NYT article also had the effect of increasing the regular level of traffic to the site, to an average of 141 visits per day. The number of visits did not regularly drop below 100 until 19 January, and from then to 8 March the site rarely received fewer than sixty per day. In short, the publicity derived from the NYT article provided a level of traffic and an audience of potential volunteers which it is hard to see how we would have otherwise reached.[24]

The Transcription Desk has been visited by users from ninety-one countries (Figure 8); most visits over the six months were from the United States, with the UK in second place.[25] This again reflects the NYT's impact and lack of comparable British press coverage during Period Two, as during Period One there were more than double the number of visits from Britain as there were from the United States (Table 1).

Figure 9 shows the number of registered volunteers, the number of manuscripts transcribed, and the number of transcripts signed off as complete. During Period One, twenty-three accounts were registered on average each week, though this dropped to eight per week during the weeks ending 26 November to 23 December 2010. Publication of the NYT article saw the volunteer base more than treble from 350 at the end of Period One, to 1,087 when the statistics were next recorded on 7 January 2011. However, after this flurry of activity, the weekly average of newly registered users dropped to twelve, lower even than during Period One. Indeed, from the week ending 18 February to 8 March, week-on-week growth of the overall user base dropped to less than one per cent, with an average of five new registrations per week.[26]

Perhaps unsurprisingly given the project’s nature, most respondents came to hear of Transcribe Bentham through online media, with the NYT article – which was published online as well as in print – being the single most commonly cited source (Figure 10). Our approaches to the British media were largely in vain, while the American press – including the Boston Globe, the Chronicle of Higher Education, as well as the NYT – reported on the progress of the project much more enthusiastically. We did not send out a press release in the United States or in Europe; the journalists who reported on Transcribe Bentham in the press and on the radio approached us for information, and in this regard, the blogosphere seems to have been an important means of attracting media attention. Other than sending out additional press releases in the UK, it is unclear what more we could have done to promote the project to the British media; it is also worth noting that historically, the Bentham Project has had trouble gaining media attention for Bentham studies. That said, the British press were not entirely unreceptive: an article on academic crowdsourcing, which included a report on Transcribe Bentham, appeared after the testing period, in the 11 September 2011 edition of The Sunday Times [Kinchen 2011], while another small piece appeared in the November 2010 edition of Wired for the iPad. It appears that it was traditional media in the United States – albeit their web presence – which was the most important recruiter of volunteers.

Google Adwords was a failure for us as a recruitment strategy. Our advert was displayed 648,995 times, resulting in 452 clicks, but sent no visitors to the Transcription Desk. The team’s lack of experience in using Adwords may account for this failure; regardless, it was considered too costly an experiment to persevere. Social media such as Twitter and Facebook, while raising awareness of Transcribe Bentham, also appears to have had little impact in driving traffic directly to the site, despite staff using them on a routine basis for publicity, communicating with volunteers, and issuing notifications.[27] Twitter, in particular, appears to have been more a means by which technology-savvy users have followed Transcribe Bentham's progress, rather than acting as a recruitment method. Nevertheless, social media has helped to integrate the community to an extent, by providing a platform for editors to share results and keep volunteers and other parties informed.

Time and budget constraints impacted upon the success of the publicity campaign. We spent around £800 of our budget, though the campaign cost more overall, as we used additional funds acquired through a UCL Public Engagement bursary and other sources. With more funds we might have been able to try different strategies, and a commercial advert in a history magazine might have been possible. It also took time for the press release, distributed at the launch of the Desk in September, to yield results. Project staff liaised with journalists soon after the launch but, owing to the production process, stories often did not appear for some time after. Wired published its piece in November while The New York Times article appeared on December 27th – unfortunately when project staff were on holiday and unable fully to capitalise on the publicity or deal with enquiries immediately – some months after the launch. Thus, the user base only started to grow, and regular participants emerge, as the project’s funded testing period moved into its final three months.

Analysis of user survey responses and user profile pages suggests that most of those interested in taking part in Transcribe Bentham were educated professionals (Figure 11). Ninety-seven per cent of survey respondents had been educated to at least undergraduate level, and almost a quarter achieved a doctorate. Almost two-thirds of survey respondents were female,[29] whereas, for example, those taking part in Galaxy Zoo were overwhelmingly male [Romeo and Blaser 2011] [Raddick et al 2010].[30]

Over a third of respondents were either academics or higher education students of varying levels, while those working in the arts, editing, and IT also registered with the project (Table 2). Almost a tenth of respondents were retirees and – from information gleaned from their user profiles – it appears that at least two of the most active transcribers were retired, while the most prolific volunteer had taken a year out from study. None of the regular volunteers were university students, and nor were any school pupils or teachers; the outreach strategy to London schools thus failed to generate sustained participation. Only one school responded to the letter and, while the Raines Foundation School showed enthusiasm at their visit to the Bentham Project, there was no sustained activity on the site. With an enhanced publicity campaign and more time, we would have been able to expand our outreach to schools by targeting independent schools, which may have had more resources available to dedicate to such a project.

Holley found that while the age of crowdsourcing volunteers “varies widely”, the most active were mainly “a mixture of retired people and young dynamic high achieving professionals with full-time jobs” [Holley 2010]. Our results generally agree, though our survey results would suggest that Transcribe Bentham volunteers were, generally speaking, perhaps a little older. Most respondents were at least forty-one years of age and, notably, a fifth were over sixty. Where it is possible to glean information, the most regular transcribers appear to be in these upper age brackets. These results emphasise, at least for a project like Transcribe Bentham, the importance of recruiting users with plenty of disposable free time, and those associated with academic and/or professional backgrounds. Our publicity outreach to these audiences was, therefore, worthwhile.

Only a minority of respondents had worked with manuscripts, or had any specialised training in reading historical handwriting, prior to taking part in Transcribe Bentham.[32] There is thus little sense that Transcribe Bentham, in spite of publicity efforts to engage that circle, tapped into a pool of experienced and/or trained historical transcribers. However, a handful of respondents did have some prior experience, and they included a medievalist, a historical editor, and one particularly experienced user who had previously transcribed 25,000 documents dealing with the Canadian government’s relationship with Aboriginal Canadians.

Assuming that all 7,441 unique visitors to the Transcription Desk (as recorded by Google Analytics) were indeed individuals, then only a maximum of six per cent of those who visited the site were moved to register an account. Yet the volume of work done by relatively few was formidable indeed. 439 manuscripts were transcribed during Period One at an average rate of twenty-three per week, though only one volunteer then submitted work on a consistent basis. Even after the enthusiasm of September and October gave way, an average of eighteen manuscripts per week were transcribed during November and December, though admittedly parts of the latter month were very quiet.

Transcribe Bentham really began to fulfil its potential in Period Two as media attention paid dividends, though sadly just as the funding period was coming to a close. 187 manuscripts were transcribed between 27 December 2010 and 7 January 2011 alone, an increase of 43% on the end-of-Period One total. During Period Two, seven volunteers transcribed substantial amounts of material on a regular basis, producing an average of fifty-seven transcripts produced each week until 8 March.[33]

Over the testing period as a whole, volunteers transcribed an average of thirty-five manuscripts each week[34]; if this rate was to be maintained then 1,820 transcripts would be produced in twelve months. Taking the complexity of the task into consideration, the volume of work carried out by Transcribe Bentham volunteers is quite remarkable (Table 3). Some manuscripts are only a few words long, while others can be up to two thousand words in length, and we estimate that the average manuscript is around 500 words long (plus mark-up). Transcribe Bentham volunteers transcribed over 500,000 words (plus mark-up) during the six-month testing period[35]

At the end of Period One, 53% of all transcripts were judged to be complete, but this dropped to 42% early in Period Two as large numbers of new users began to transcribe and grappled with Bentham’s handwriting and style, and the transcription interface.[36] The proportion of completed transcripts remained around this level until late January, when an “Incomplete Folios” category was made available to provide volunteers with the option of working on transcripts requiring improvement. By 8 March, 55% of transcripts were completed and locked.[37]

As Romeo and Blaser note, “[u]nderstanding the motivations of online volunteers is an important way to inspire and sustain participation”  [Romeo and Blaser 2011]. Those responding to our survey reported that they were mostly motivated to take part by a general interest in history and philosophy, in crowdsourcing and the technology behind the project, and in Bentham himself (Figure 13). A significant number cited contributing to the greater good by opening up Bentham’s writings to a wider audience as a motivation, and several even found transcription fun! Though our survey sample is relatively small, similar findings by others examining participant motivations suggest that the results are sound. Peter Organisciak found that primary motivational factors included subject matter, ease of participation, and a sense of making a meaningful contribution, while the top five NLA text correctors cited their enjoyment of the task, interest in the material, recording Australian history for posterity, and providing a “service to the community” as their main motivations for taking part [Organisciak 2010, 83–88] [Holley 2009, 17–18].

Transcribe Bentham consciously implemented means to encourage volunteers through friendly competition, or what Romeo and Blaser describe as “game like mechanics”  [Romeo and Blaser 2011]. The “Benthamometer” illustrated the project’s overall progress, and the points, multi-tiered ranking system, and leaderboard provided a visible recognition of work and progress. Other crowdsourcing projects have considered such features to be important, and it was thus surprising that Transcribe Bentham volunteers regarded “competition” and “recognition” to be of such low importance. Organisciak likewise found points systems, achievements, and leaderboards to be of secondary importance to volunteers [Organisciak 2010, 90].

However, this finding comes with a significant caveat: the most active super transcriber was indeed partly motivated by both competition and recognition, and it is clear that projects must be flexible enough to meet the motivational needs of a variety of volunteers.[42] Perhaps others were subconsciously competitive, as another volunteer noted that “I’m usually a very noncompetitive person but […] became inspired to do more when I find I’m losing my ‘position’” [on the leaderboard] – it’s ridiculous! but quite fun”.[43]

Many respondents found the intellectual challenge and puzzle-solving aspect of transcription enjoyable and highly rewarding, with several comparing deciphering Bentham’s handwriting to solving a crossword or sudoku. Two transcribers were excited by working on manuscripts which may not have been looked at since Bentham wrote them, while contributing to the accessibility of the digital collection and the production of the Collected Works was also a factor for some.[45] One volunteer notably described Transcribe Bentham as a “literary form of archaeology”: “Instead of using a brush to uncover an object, you get to uncover historical information by reading and transcribing it. It leaves his legacy available for all to access.”

This is an important point, and supports Holley’s suggestion that volunteers are more likely to take part in projects run by non-profit making organisations, as opposed to those run by commercial companies [Holley 2010]. When volunteers submit transcripts to Transcribe Bentham, they agree to the following disclaimer: “You are contributing to an Open Access initiative. For the purpose of UK legislation, copyright for all materials within Transcribe Bentham: Transcription Desk, including this transcription, resides with UCL (University College London). If you do not want your writing to be edited and redistributed at will, then do not submit it here.”

Overwhelmingly, survey respondents had no problem with this, which speaks to the mutual trust and respect between a project and its volunteers which is vital for success. Users recognised that UCL had made freely available priceless manuscript material, a transcription interface and instructions, and that their work was a transcript of Bentham’s writing rather than original intellectual content. Such approval did, however, come with the proviso that the submitted transcripts would not be commercially exploited, and would remain accessible to all in the future; the free availability and long-term preservation of the transcripts in UCL’s digital repository certainly meets these requirements.

The vast majority of those who visited the Transcription Desk did not become active users, and it is thus just as important to understand what dissuaded people from participating as it is to understand what encouraged them (Figure 14). Some were daunted by the task, or felt that their contributions might not be of sufficient standard to be deemed worthy. For example, one respondent noted that he had been unable to read Bentham’s handwriting “with enough clarity to be happy recording my guesses in an important academic project,” while another “confess[ed] to being a bit intimidated […] fear of failure?” This is a shame, as any contribution to Transcribe Bentham is beneficial to the project; perhaps in the future the project team could provide more reassurance to volunteers that all efforts are valuable. Another respondent decided that she could not take part because “I need a paying transcription position”; Transcribe Bentham is in no position to offer remuneration, and – as discussed above – it was considered that offering money for participation would be contrary to both the project’s collaborative principles and spirit.[46]

It is notable that no survey respondents mentioned the importance of feedback from project staff or prompt responses to submissions. However, we have discovered through correspondence and observation that volunteers greatly appreciate notifications as to whether or not their submissions have been accepted, as well as responses to email requests for further information. Indeed, volunteer enthusiasm can noticeably fall away when feedback and acknowledgement are not given, and we may have lost potential long-term contributors during the December break when staff were away. This suggests that project staff must devote time to answering queries from users and to maintaining social media features in order to facilitate interaction, otherwise the project can appear dormant, and volunteers may lose interest or feel disconnected and exploited.

The single most important dissuading factor, however, for those who responded to the survey, was time, or rather the lack of it, in which to transcribe and encode, or learn how to do so. Reading and deciphering the manuscripts was found to be the main challenge: as one respondent replied, “[t]he real difficulty is in reading [Bentham’s] handwriting”. Indeed, over half of respondents found that deciphering Bentham’s hand took longer than encoding. Though text-encoding was an additional complication to the process, encouragingly few survey respondents found it prohibitively difficult.[47] However, this comes with the significant qualification that almost two-thirds of respondents found that it took some time to get used to the encoding process, and that a significant proportion thought they spent as much time encoding as they did deciphering Bentham’s handwriting.[48]

Indeed, most free-text responses noted that encoding (and the extensive instructions on how to do so) were daunting to beginners, but that the process became reasonably straightforward with practice.[49] Several respondents were not so sure: one believed there was “[t]oo much markup expected”, another that encoding was “unnecessarily complicated”, and one – who, unsurprisingly, is not a regular transcriber – found encoding “a hopeless nightmare” and the transcription process “a horror”. To many, text encoding was simply a time-consuming complication and almost certainly dissuaded participation. As one respondent put it, “The transcription process has been a little more complicated than I anticipated. I thought I would be able to type away but I have to learn a little more before I get started, so I’m not wasting anyone elses [sic] time.”

Some proved very able at text-encoding, to the point where almost half of survey respondents were confident enough to type at least some tags manually rather than use the toolbar. However, the negative views of encoding are very real concerns which must be addressed when improving and refining the transcription tool.[50] Indeed, the majority of survey respondents were unfamiliar with the principles and practice of text encoding before taking part in Transcribe Bentham, though with a few exceptions: some were aware of the basics through editing Wikipedia, writing HTML, or using early word processors. One volunteer in particular noted, that “I’m part of the wiki generation -- we have encoding encoded into our blood”.[51] While crowdsourcing projects should not underestimate the IT proficiency of their volunteers, or their ability to learn new skills, the task at hand should be made as simple as possible.

As such, we made the transcription and encoding process, and the instructions, as user-friendly as we could given our requirements and our short time-frame. However, these survey responses and the project’s inability to retain all but a few long-term volunteers suggests that the task appeared too complex and time-consuming for many prospective volunteers. Though we can offer guidance and assistance to those attempting to decipher Bentham’s handwriting, evidently, simplification of the transcription process and instructions would improve user recruitment and retention. For example, the introduction of a What-You-See-Is-What-You-Get interface, as an alternative to the transcription toolbar, with encoding occurring “under the hood”, would prove an attractive option for many volunteers. They would then be able simply to concentrate on transcription without having to digest lengthy instructions [Causer, Tonra and Wallace 2012].

Fostering a sense of community would, we hoped, stimulate participation by encouraging users to strive to achieve common goals. Our community building strategies, however, appear to have had limited success. For example, only one pre-existing transcriber attended the public outreach events held in May 2011. This transcriber did appreciate the opportunity to meet project staff and experience face-to-face interaction. Of course, the volunteers who live outside of the UK were inhibited from attending these events and thus were unable to integrate physically with the community. In terms of fusing the Transcribe Bentham community together therefore, this particular programme of public events was ineffective.

Furthermore, despite the project having integrated social media facilities into the Transcription Desk, there is minimal evidence of interaction between Transcribe Bentham users. We are only aware of a handful of occasions on which users have communicated publicly with each other via their user page, a transcript, the discussion forum, or the project’s Facebook page. While users may have exchanged private messages or emails, the survey results suggest otherwise: only one respondent reported that she had contacted another via their user page, two had done so via the discussion forum, and another via Twitter.

This is surprising, as it was anticipated that users would demand such social functionality. Indeed, the North American Bird Phenology Project's survey respondents specifically requested a discussion forum to communicate amongst themselves, and the NLA was met with requests from text correctors to provide a facility by which users could contact each other [Phenology Survey 2010] [Holley 2009, 23]. Many of the major crowdsourcing projects have discussion forums in which volunteers are active to one degree or another.[53] These are certainly busier than the Transcribe Bentham forum which – rather than being a space for volunteers – became merely another avenue by which users contact project staff or in which project announcements are made. Perhaps a larger volunteer community is required for the forum to be used to a greater extent.

Moreover, by the end of the testing period, seventy-eight per cent of survey respondents stated that they had not added any information about themselves to their user profiles.[54] A cursory look through registered users’ pages – even those belonging to super transcribers – reinforces the impression that volunteers are not particularly interested in social media facilities, or perhaps even being part of an online community per se, as hardly any contain any information beyond a name and, in a few cases, a photograph or avatar. Survey responses suggest why this may be the case. One respondent had “added minimal information because I don’t know enough about the community […] to want to make my information available to others”. Others were concerned that such information “seems unnecessary”, that they were “[n]ot sure how this would be helpful”, thought personal information would be “more interesting and useful to the T[ranscribe]B[entham] organization than to individual contributors”, or didn’t see the feature as important. Two further respondents expressed doubts about the wisdom of putting too much “personal profile info on public sites”, which may be a significant issue for future crowdsourcing projects seeking to create a community. Furthermore, while there have been technical issues with the site’s social features, transcribers do not appear too concerned about these problems as we have received no enquiries about their non-functionality.

On the other hand, some wanted to know more about their fellow volunteers. One believed that “it is important for each person involved in the project to at least let us know their educational background and philosophy. It encourages discussion”. Another thought that it “is only possible to create an online community if people are willing to say something about themselves”. However, they were in a definite minority, with most seemingly uninterested in, or unaware of, the site’s social media capabilities.

Transcribers have proven wholly good-natured in going about their work: there have been a couple of easily-resolved but unintended revisions of others’ work, but certainly no evidence of “edit warring”, repeated reversions, or arguments. There is certainly none of the overt territoriality exhibited in the tagging of text in one NLA newspaper, when an anonymous user requested a registered text corrector to “please refrain from appropriating and/or inserting any [text] correction or tag” concerning a particular historical figure, whom the anonymous user appeared to believe belonged to him or her.[55]

That said, there is no evidence that Transcribe Bentham's volunteers have collaborated directly upon manuscripts; rather, they participate in what might be termed unconscious collaboration, with volunteers sometimes adding another layer of work onto what went before. As one respondent to the survey suggested, it “[s]eems to me the whole system is a collaboration, since we’re encouraged to submit unfinished transcriptions & others can work on them. Maybe a serial collaboration”. Two respondents were unaware that direct collaboration was possible, while another did not “feel secure enough with coding to muck up someone else’s work”, and presumably worked instead on unedited folios.

Indeed, three-quarters of edited manuscripts were worked upon by only one volunteer, while relatively few manuscripts were transcribed by three or more. Those manuscripts transcribed by multiple volunteers were generally begun with by one or more new users and otherwise “abandoned”, and later completed by a more experienced transcriber. This all suggests that volunteers appeared to prefer starting transcripts from scratch, and to work alone (Table 6), with communication and acknowledgement from staff being of much greater importance than collaboration with other users. This might appear counter-intuitive, as dabbling with partially-completed transcripts could allow volunteers to get used to the project and transcription process, and be less daunting than starting a transcript from scratch. The advent of the incomplete folios category in mid-January 2011 did see regular volunteers finishing partially-complete transcripts, though this option has not been popular with new volunteers, who seem to prefer to start with an untranscribed manuscript. Indeed, we have regularly received complaints from volunteers about being unable easily to distinguish between untranscribed, partially transcribed, and completed transcripts. Perhaps registration acts as a barrier to those who might wish to tweak a little bit of encoding or transcription here and there.