Unlike the apocryphal tales of teenagers developing million-dollar apps in an evening hack-a-thon, the thought of building your first mobile app research tool should seem like a daunting task, because it is. Such a project can overwhelm you and your team, as it has ours at times, especially if not managed within the scope of your expertise and resources. The larger projects we mentioned earlier involved multi-year endeavors with dozens of professionals, more than 100 volunteers, and more than $100,000 in grant monies to produce. That said, we also did not need to build such enormous programs to test many of the ideas that we eventually started focusing upon, including most of our recent studies. The bigger projects had numerous and varied goals, including civic engagement, service learning, and even some exploratory meandering.

For the most recent studies, though, we simply built spare formulaic apps that had the functions we wanted, without any aesthetic flourishes. We then installed those apps onto a set of identical tablet computers dedicated to this research. That refined approach avoided the complications we have had with the development and installations of the other apps, and various problems with compatibilities to the many different handsets that could be using the apps. The smaller and more-focused apps took roughly a day for a professional coder to complete, and even less in cases in which existing code could be reused in similar ways. So the hack-a-thon approach can be fruitful, as such, if focused.

Yet that does not mean these more-limited efforts were cost-free. A full day of paid mid-level programming typically costs about $1,000, and that does not include the pre-planning, post-analysis, or fixing the errors that become apparent during the design process. If the app design is simple enough, and if the scholar has some programming experience, an app of this nature probably could be made in a week or so by hobbyists familiar and comfortable with basic HTML5, CSS 3, and JavaScript. The initial design work of the throw-away app – including how it would function, and precisely what media would be delivered when – was roughly the same amount of work as any other equivalent research design. Instead of printing and handing out surveys, for example, that same data was gathered and delivered digitally through the app. In the past 15 years, collecting survey data online has become the norm, and this merely represents, from our perspective, the next iteration of taking digital data collection into the field.

Publishing apps to the App Store or Google Play market requires another level of expertise and more time, but Android APK files (the program installed on the devices) can bypass the markets altogether and be shared directly via email. As an open system, making, testing and distributing via Android can be relatively easy (although it creates different sets of issues related to the multitude of devices that operate on Android OS). Apple has some work-arounds at this point, too, such as HockeyApp, which allows non-store distribution. If the goal is to conduct research on specific devices that you can reach via email, rather than a mass market reached through the app markets, then a lot of the hassle of publishing and sharing apps can be avoided simply by focusing on Android development or working through services such as HockeyApp.

As digital-media delivery systems become more and more accessible and open to the general public, mobile app development is an example of an activity within a ubiquitous system that many scholars could navigate within and directly participate. While reachable, the bar still remains. Scholars without adequate coding skills and/or resources to hire a programmer could tailor the content of readily available third-party software and test through that system, but then the researcher would have limited access to the data collected and to the controls of the overall app environment, which we consider significant trade-offs. Large-scale public projects, such as Fort Vancouver Mobile (www.fortvancouvermobile.net), create several research paths, but they also have significant drawbacks as well, including the exertion of enormous amounts of energy, often without the equivalent academic credit in many institutions, and with continual maintenance concerns. Starting small, with a focused research tool, and taking that mobile app all of the way through the implementation process, before raising ambitions and the stakes in sustainable increments, is the path we recommend as a lesson learned. Building a mobile app also means becoming aware of the many possibilities that mobile technologies offer. Scope creep and feature creep are ever-present. Focus is difficult to maintain, with so many available options, and easily reachable enticements, along the lines of, “We can do that, too,” which easily can derail original intent or distract from the research purpose and drain resources, of which the needs already are many.

This broader challenge was one we could have anticipated, of course. Trial and error remains a common element of research and development procedures, as well as of collecting data in new ways through emerging protocols. But the details of these difficulties were revealed in the discovery process, not in the abstract conversations about what could go wrong. For example, we initially used both mobile phones and tablet computers for our field tests. After finding no significant differences between them in regards to how people reacted to our research app (data from our first collection confirmed this), we used tablet computers exclusively in ensuing field work. The bigger devices were easier for us to track in the field, provided a larger interface for the participants, were not as likely to be pocketed and stolen, and also importantly, were easily accessible to us in terms of equipment with which we had access and were familiar. We encountered a few minor issues regarding the alteration of survey items as we planned the study; rather than having the ability to directly change the items, for example, working through a developer bogged the process a bit. Working with trusted development partners, though, cannot be overstated. We hired and worked with a few different developers before and during this project. The ones who did not deliver sapped our resources – including time, money, and enthusiasm – but the highly valuable people who did seemed to be able to nearly perfectly translate our ideas to the technologies and in the process generate new forms of research instruments.

Mobile technologies can gather deep data about a person, doing real tasks, in real situations, in real settings, but that data – even if it is considered “authentic” – is not complete, and it is not highly controlled. Mobile apps can acquire extremely detailed data from people unobtrusively, in the flow of life, and in situ. With our apps, for example, we have tracked from the exact moment a user opens a program until the user closes it, and everything in-between. These acts included the precise time and location of the person when any sensor, such as a touch of the screen, is triggered. That geospatial context – directly connected to specific interactions and users, in particular places at exact times – offers unprecedented potential for the granularity of data collection and data-point triangulation.

That insight is arguably the most meaningful addition to our understandings of the research topic, but problems arise when researchers, including us, begin to parse that data, or combinations of that data, into usable units, in terms of determining exactly what variables are affecting what behaviors. In short, one cannot possibly control for all of the variables in “the wild.” Action Research with mobile devices, though, can create fascinating forms of self-contained triangulation. The devices gather detailed analytics of use, and then they also can push users into traditional research instruments, with more methodological certainty, such as Likert-like scale surveys, which are geolocated and taken in the context of the other data points. If a researcher observes the test subject performing the tasks with the app and taking the survey, another perspective dynamically is added to the rich mix of app-collected data.

We also have found that people generally enjoy participating in research that does not feel like research. We typically have kept our hypermediate interventions, such as pushing a survey to the user, to a minimal time (ideally 5 to 7 minutes). As such, we tend to ask just a few questions at a time, which has reduced the traditional data collection results, in exchange for the instrumental transparency of in-flow data of ordinary use. The trade-off comes in the form of a high-quality sample composed not of a homogeneous group (e.g., students, members of an organization), but of a wide-swath of the surrounding community. Too often, in our opinion, validity gets sacrificed for statistical “purity.” This App-Maker Model approach allows for both, though the breadth of some statistics is lost in the pursuit of a more-representative sample.

An authentic real-world data sample is one drawn from the complexities of humanity in action in authentic settings. Laboratory scientists, reductionists, and those who favor highly controlled experiments, of course, might be troubled by the inherent messiness of this style of data gathering (and have told us so in various formal and informal settings). The App-Maker Model approach means looser levels of control compared to traditional laboratory experiments. In a lab, we can put people into evenly divided groups, in the same environment, and examine their behaviors within a consistent experience. In the field, conditions are harder to manipulate (though our randomized app conditions, more or less, address this), environmental factors such as sunlight, noise, and temperature can impact the results, and the wider swath of humanity studied requires more adept communication skills and responsiveness to the situation than, for example, interactions with homogeneous crowds of college students in a lecture hall.

In addition, the mass of data collected by mobile devices could be misleading, or could be misinterpreted, without other methodologies to triangulate and fully comprehend the results. People might not naturally cooperate in pursuing the goals of the research agenda. They might not stay on task. They might act in ways unaccountable to the device, and irrational, such as leaving an app turned on and running, so it appears to be in use, while instead the person has placed the smartphone into a pocket and forgotten about the app. “Constant use” could be mistaken for neglect.

We can anticipate some of these issues, but they also often come in forms we do not expect and don’t discover until in the midst of the situation. To us, those experiences just deepen our understandings of mobile media. For the most part, as an example, we have had pleasant weather during our tests, and everything has proceeded according to plan. On days we didn’t (during a couple of early prototype testing periods), rainy days turned highly organized and resource-intensive testing environments into wastelands of resources. From those failures, we learned to constantly check the weather reports. We also learned how environmental conditions (such as nearly unavoidable screen glare caused by the sun at high noon) significantly could affect use. In another testing situation, we had a train park next to a collection site less than an hour into the event. It proceeded to backfire at irregular intervals for the duration of the data collection. Many people had trouble hearing the multimedia portions of the app (and rated their experiences accordingly). The noise held constant across conditions, and although the raw numbers experienced some differences, the distinctions between groups we investigated likely did not. During another collection, the cold weather impacted people’s abilities to effectively utilize the touch screen (apparently cold fingers do not register on a touch screen well), a problem that potentially could have been mitigated with a stylus, if we had thought to bring one. Ultimately, these observances constitute a real view of the environmental factors that people face in the wild and help remind us to take these things into account in the design process.

The last and perhaps most important issue we faced involved connectivity, as most outdoor spaces do not have accessible Wi-Fi. Our field experiments at Yellowstone National Park, for example, have been severely constrained by connectivity issues. During our initial data collection with the one-time-use app, for example, we relied on a cell phone data plan, which became erratic. For ensuing studies, everything was stored on the device, allowing the media to load quickly, and data remained on the local device until we downloaded it to a common file. In studies such as ours (working in a park with less connectivity), this higher level of control allowed for a consistent and expedient experience. Apple has improved its ability to host local apps with the HockeyApp application, but we found the Android APK to have more robust features and less glitches. So we would recommend staying in an Android development environment for initial pilot explorations, at least at this point in history.

Conducting Action Research within mobile environments offers solid methodological choices for many situations. The App-Maker Model can uncover and build knowledge that other approaches simply cannot reach. It has been potent, in particular, for our studies of the medium, media forms, interaction design, remediation, and mobile learning. We have used mobile technologies to track behaviors in traditional HCI contexts, such as measuring time on task, in design contexts, such as A/B comparisons of interface features, in learning contexts, such as with error rates, and in broader medium and media studies, such as comparing knowledge transfer from a brochure versus a mobile app, and response to information in video form versus audio-only. Further, we have made new media forms, such as location-oriented videos, and studied how audiences reacted to them. We have explored prompts and triggers for audience interaction, particularly those in which users might be compelled to make and share media. All of those different approaches have been enabled by the opportunities afforded when researchers build their own mobile app instruments, which, in turn, creates some forms of experimental controls.

By pioneering in this realm of mobile technologies, and actively participating in its construction, through building in it and seeking out new areas to explore, a researcher can break ground and set the foundation for a field or subfield. Such unexplored territory has significant value, and the findings can feel thrillingly fresh. When we have built mobile apps, and have seen what they can do, we often have small moments in which we truly think we might be the first people on the planet to have encountered a particular scholarly research situation. New mobile affordances and applications emerge continually.

Digital humanists have to ask digital-humanist questions in these cases, because no one else is going to do that for us. Generating an original thought about that sort of situation can be an exciting moment, and it happens relatively frequently through this kind of formative research. Those moments, in turn, can open new research paths of all shapes and sizes. As a research engine, this approach can reveal many otherwise obscured issues. Ideas for next steps appear everywhere. Those recurring moments empower and energize, and research feels less like incremental boulder rolling and more like an adventure.

Action Research through mobile technologies also can be ineffective and inefficient for some research pursuits. This App-Maker Model approach, by nature, is time consuming and involves laborious cycles of planning, action, reflection, and so on. Combine that pace of research with the tedious nature of app development (and publishing), and some researchers simply might not have the patience or perseverance, or the interest, to see what eventually might come from such long-term efforts.

Many other kinds of research methodologies have been developed over decades and proven themselves in specific niches for answering certain kinds of questions about particular kinds of things. They also have developed a methodological certainty that comforts many researchers, administrators, and publishers. Frankly, they are less trouble to use. Action Research, by nature, might be more exploratory and also more unpredictable than other types of methodologies. The App-Maker Model approach also integrates the researcher directly into the research, rather than keeping operations at arm’s length and safe from direct influence by stakeholders’ desires, even subconscious ones. Sometimes, such observational distance helps provide clarity. Likewise, such detachment reveals flaws that otherwise could be missed, especially by researchers invested in the production process.

Risks also can lead to failures, sometimes in grand fashion. The research community generally wants new ideas and research paths to emerge. Contradictorily, it often has a difficult time dealing with new paradigms and findings that challenge the status quo, or scholarly traditions, in terms of fitting those into pre-existing conventions and forms. Action Research, for example, has been a widely discussed and supported scholarly approach for decades, yet its messiness and methodological uncertainties keep it on the fringe of the academic publishing world. Work with this approach can be widely unappreciated, or underappreciated, and confusion about its standards and techniques contribute to its outlier status. When people in power do not understand this approach, or dismiss it as unscientific, there are ramifications beyond just social capital. By not following the specific leads of others, down well-trodden methodological paths, action researchers can be ostracized or even punished, especially when issues of promotion and tenure are raised, especially by non-digital-humanists.

In our experiences, for example, we have had research grants from the National Endowment for the Humanities indiscriminately classified as “service” work because they funded these sorts of Action Research initiatives. Besides the various issues related to credit for the work, formative studies in any new field, like this, can get lost in search of landmarks, because of the novelty of the technologies and the inexperience of the test subjects (and researchers) with those technologies. That should not be a surprise or be set up for organizational punishment. This is part of the wayfinding, as getting lost can sometimes lead to interesting, otherwise unknown places. In the meantime, working with technologies that change and rapidly update creates a churning of effort and resources needed just to maintain a place within the research realm. When operating systems are updated, when hardware innovations appear, when societal use patterns dramatically shift as the newness of an idea wears off, action researchers need to remain nimble and ready to adapt. Otherwise, they quickly get left behind, and their work could become irrelevant. Those recurring moments of shifting and reconfiguration can be draining and exhausting.

Mobile technologies allow data to be collected in novel ways. During our recent research, we have combined traditional ideas with new mobile techniques, such as inserting survey questions into an app, so that they are interspersed throughout the experience, rather than solely being collected at the end. While the survey itself is standard (e.g., asking for basic demographic information, using Likert-like scaled questions to better understand responses to the mobile intervention), the process is fundamentally changed by the technology’s affordances. For example, the questions can be automatically randomized. This gives each user the items in a different order, helping to eliminate any fatigue effects that the ordering might cause. The data is entered directly into the app, limiting data entry mistakes to the ones created by users, and the app exports the data into a common database form. The surveys also can keep the user from advancing through the app until all of the questions are answered, eliminating missed questions, and preventing users from skipping parts, keeping the survey data complete. In turn, the data can be collected in ways that simply have not been readily available to researchers in the past. For example, as soon as the research app has been opened, if the privacy permissions have been allowed, the test subject will create a complete record of all interactions with the mobile device when using that app, including a GPS trail of where those interactions took place, and a time and date stamp. The richness of such available deep field data is unprecedented.

Yet the more data collected from a test subject, the more choices that have to be made about how that data is collected, how that collection process affects the data, and what the data inherently means. In short, the potential of what we can do now dramatically increases the possibilities and therefore exponentially expands the complexity of what can and could be done. We do not know yet, for example, how survey data collected in a mobile fashion, within an app, and as part of a mobile experience, affects the ways in which test subjects answer the questions and approach those surveys. What would the results be if a paper survey was included in the same experience, in place of the mobile survey, and given at the same time? Other data collection techniques are more complicated and need more consideration, whether they are extensions of traditional methods or some sort of new methodology emerging from the primordial technological ooze. In turn, this data resembles discovering a new species of information. We think we know something about it, but we do not know enough to make many assumptions without carefully measuring and tempering our leaps. In the process, we also encounter the risk of becoming too reliant on the data and the machinery and losing mindfulness about what we study and what the data reveals.

This freshness of perspective was the most predictable benefit, in our experience. The different types of data allow for new ways to triangulate findings, and provide ways to validate the data analytically beyond statistical p values. The biggest drawback here goes back to Issue 3; collecting data in a new way does entail risk and doesn’t fit into the prescribed patterns most people are trained to recognize and evaluate in graduate school. Combining a humanistic approach with qualitative and/or quantitative data is historically rare, but scholars are beginning to integrate these approaches more frequently [Kinshuk, Sampson, and Chen 2013] [Lynch 2014] [Spiro 2014]. A journal editor, in turn, could face significant challenges finding appropriate reviewers to which such an article can be sent for feedback, and junior faculty, eager to build publishing credentials, hardly can afford additional obstacles in an often highly challenging and ultra-competitive peer-review environment.