Students’ Behaviour Playing an Online Information Literacy Game

This paper describes how college students played the web-based BiblioBouts Information Literacy (IL) game, which ushers players through the library research process while they complete a research-and-writing assignment. The game teaches students basic IL skills including creating citations, judging citation completeness, assessing author expertise, assessing source relevance and credibility, judging quality, and assessing accuracy. BiblioBouts’ collaborative and social mechanisms help students leverage their own research efforts in finding sources, evaluating their usefulness, and choosing the best sources, with their classmates’ efforts so that everyone benefits. Players benefit from receiving hands-on practice and experience with the wide range of information literacy (IL) skills that confront them during the process from conducting library research to completing writing assignments. Both quantitative and qualitative game-play data were gathered from game-play logs, game diaries, focus group interviews with student game players, and personal interviews with instructors. These data were analysed to determine typical game-play styles, how long students played the game, and the impact of scoring on the way the students played the game and engaged in IL activities. The results were used to improve game mechanics and player engagement. The R&D team’s experience building an online, interactive IL game demonstrates that game design must first focus on evaluations of player behaviour followed by game-system improvements that are expected to affect the desired gameplay behavior. The BiblioBouts game presents an innovative method for learning IL competencies and is unique in its social, collaborative, and interactive approach to educational gaming. It is hoped that this article will encourage IL librarians to explore games and other alternative forms of IL instruction.


Introduction
Information literacy (IL) is a critical 21 st century skill which empowers students to think critically when locating, evaluating and using online information sources.While these skills are universally seen as crucial to student success in academics and professional life, as well as forming the basis for lifelong learning, research repeatedly shows that many of today's university students enter the university without having received any IL training.Academic IL programs have ambitious goals; however, only a minority of http://dx.doi.org/10.11645/5.2.1637 institutions feature first-year experience programs where information literacy content is mandatory (Boff and Johnson, 2002).Although faculty delegate teaching students information literacy skills to librarians, faculty are primarily concerned with disciplinary coverage, and are therefore reluctant to cede valuable in-class time to librarians (Hardesty, 1995;Breivik, 1998;Hrycaj and Russo 2007).Students are intent on reaping the rewards faculty and library staff give them for mastering the core concepts of the academic disciplines and focus on their coursework rather than on information literacy skills and concepts.Librarians fight an uphill battle, trying to reach as many students as they can through a wide range of venues such as workshops, short courses, virtual reference assistance, web-based instruction pages, and walk-in assistance at information desks.
To address these challenges, a research & development (R&D) team at the University of Michigan's School of Information has created BiblioBouts, an online, in-class game to teach incoming undergraduate students information literacy skills and concepts.Funded by a four-year National Leadership Grant from the Institute of Library and Museum Studies, BiblioBouts has been field tested in 13 classes at 6 universities during the process of development.The team is comprised of library and information science faculty and graduate students, computer programmers, web designers, and librarians at Michigan and at four participating academic libraries in the U.S., who built and tested the game through an iterative process of development.The authors of this paper are team members responsible for the evaluation of BiblioBouts.

Literature review
The literature on games for learning is extensive, covering many educational disciplines and types of games.However, the literature specifically addressing games for IL education is significantly limited.In the broader literature, several authors have pointed to the correspondence between structural elements of game design and principles of good learning.Rather than dismissing games as frivolous or distracting, these researchers have explored the cognitive and pedagogical potential of games.For example, Johnson praises games for their ability to help us "find order and meaning in the world and make decisions that create order" (2006, p. 31).Gee argues that "games are potentially particularly good places where people can learn to situate meanings through embodied experiences in a complex semiotic domain and meditate on the process" (2003, p. 26).Of his 36 Video Game Learning Principles, several are embodied in BiblioBouts: practice, critical learning, metacognition, probing, discovery, and just-intime learning (Gee 2003).Games are a type of problem-based learning, wherein players are presented with engaging challenges that they must solve by learning and utilizing new skills.
From this perspective, games can be seen as a cognitive playground for the safe exploration of ideas and skills.As a context for learning, games offer an alternative environment to the traditional instructionist school-based model, because they "encourage collaboration among players and thus provide a context for peer-to-peer teaching and for the emergence of learning communities" (Squire and Jenkins 2003, p. 29).In fact, games can be conceived of as active learning environments in which "students can learn by doing, by undertaking purposeful and meaningful tasks, reflect on their experiences and work with others to achieve learning goals" (Whitton 2010, p. 52).John Seely Brown argues that online multi-player games are examples of a new culture of learning consisting of a fusion between information networks and bounded http://dx.doi.org/10.11645/5.2.1637 environments of experimentation.These learning environments are designed as meritocracies which require teamwork, encourage experimentation and innovative strategies to overcome obstacles.Ideally, "the reward is converting new knowledge into action and recognising that current successes as well as failures are resources for solving future problems" (Thomas and Brown 2011, p. 88).
The effect of motivation and engagement is a significant area of exploration in the games literature.The elements of challenge, reward and feedback in games contribute to their making them fun and engaging.The potential to create user engagement in the context of learning is one of the important features of digital games (Whitton 2010).Empirical studies have shown "increased knowledge retention by those using an educational game compared to those receiving conventional instruction with lectures and paper-based materials when specific information or concepts are targeted or the game is used as a reinforcement or practice tool" (Leach and Sugarman 2005, p. 192).Since the traditional lecture format may not be effective for engaging or maintaining the interest of tech-savvy students who have grown up using the Internet, games offer a unique alternative for engaging and motivating players to "learn new skills and knowledge without realising they are in the midst of the learning process" (Leach and Sugarman 2005, p. 223).
The use of games for learning is also not without its detractors.The superficial application of game-like elements (points, badges, levels) as marketing gimmicks on commercial sites is derided as "gamification," a pejorative that suggests a cynical ploy without well-designed meaningful content.This phenomenon has also been labeled "pointsification" (Robeson 2010) and "exploitationsware" (Boghost 2011) in attempts to differentiate it from meaningful games that truly embody player choice and engagement.However, when carefully implemented and grounded in learning principles, games still hold the potential of creating new educational environments.
Library education and information literacy instruction can benefit from the potential of games to motivate and engage, as students may feel that traditional IL training is irrelevant or uninteresting in the age of Google.Martin and Ewing suggest that "incorporating digital gaming techniques into library instruction is one way to motivate and engage students throughout the information-gathering process" (2008, p. 213).The structure of repeated practice and reinforcement can be particularly effective for skillsbased learning because "games that encourage this form of iterative probing can support the development of logical thinking and problem solving, important in learning information literacy skills" (Kirriemuir 2008, p. 158).Current information literacy games do not always live up to these ambitious goals.Some games simply drill students on basic skills such as identifying an information format (book, magazine, journal), recognizing a call number, or introducing the concept of plagiarism (McCabe and Wise 2009).Others are tied to the resources of one institution (Leach and Sugarman 2005), are limited to a single discipline (Smith 2007), or are simply animated tutorials (Armstrong and Georgas 2006).By contrast, BiblioBouts is unique in its social, collaborative, interactive approach to IL skills training through educational gaming.

Structure of the BiblioBouts game
BiblioBouts is an online tournament made up of a series of mini-games or bouts, each of which introduces students to a specific subset of information literacy skills within the overall research process.BiblioBouts' setup interface enables instructors to set their http://dx.doi.org/10.11645/5.2.1637 minimum-level of expectations for students in the form of caps and quotas for each bout.Caps and quotas govern the number sources players confront in a bout.Students who exceed quotas are exposed to more sources and get more practice performing various information literacy skills.BiblioBouts' default caps and quotas are based on the expectation that most instructors want students to cite five to ten sources in their final paper's bibliography, the R&D team's experience with the amount of effort students will exert on game tasks, and the need to collect enough ratings per source (usually three to five) to enable students to determine a positive, neutral, or negative trend with regard to a source's credibility and relevance.Along with the IL content and activities that are the core of BiblioBouts, students also benefit from gaining experience in 21 st century skills such as collaboration, communication, networking, and peer feedback.Reviewing the work of other players is a crucial element of the game, as students become members of a learning community through practicing their critical evaluation skills both on their own sources and on those of their classmates, and through exposure to a broader array of research sources on their topic than they might have found on their own.Through assessing their own ratings, tags and comments as well as those of other players, students practice giving http://dx.doi.org/10.11645/5.2.1637 constructive feedback as well as receiving it.While playing BiblioBouts, students take on the role of a researcher by practicing the skills and habits of expert researchers.These new techniques and strategies will help students both in their college and professional careers.
In the Donor bout, students search the web and library databases for sources on an instructor-assigned broad-based topic and put them into play against other player's sources in the game.Players save sources and their metadata using the Zotero citation management tool.When players synchronise their Zotero sources library to BiblioBouts, Zotero passes their sources to BiblioBouts.The game awards players 100 points for each source up to quota and 200 points for each source thereafter.The default Donor quota is 8 sources but players can exceed this quota and donate more than eight sources.
The Closer bout runs concurrently with Donor plus one day.In Closer, players scrutinize their donated sources, choosing the best ones to submit to the game and making sure that the chosen sources have full citations and correct full-text attachments.They earn 100 points for each source they contribute or "close," up to the default cap of five sources which players cannot exceed.The Closer bout cap was instituted to reduce the minimum number of tagging, rating, or sorting tasks each player must perform in the next two bouts.
In the Tagging & Rating (T&R) bout, the game shifts from one's own sources to opponents' sources.BiblioBouts randomly chooses an opponent's source, displays it to the player, and asks the player to assess its usefulness in several ways.Is the right fulltext attached and is a complete citation given?What is the source, for example, is it a scholarly journal, an encyclopedia, a newspaper, a directory, or a blog?What audience does it address?Players rate the source's credibility based on their assessment of the author's expertise, the trustworthiness of the information and the vehicle in which it is delivered, and the scholarliness of information.They also rate the source's relevance based on their assessment of the usefulness and accuracy of the information and whether the content is good enough for them to incorporate into their paper.BiblioBouts encourages players to add comments telling why they gave sources the ratings they did.The number of sources that players tag and rate depends on the number of ratings per source that the instructor sets (the default is five) and the number of unique sources players closed in the Closer bout.Players earn 150 points for each source that they tag and rate up to the quota set by the instructor, and 300 points for each source thereafter.
Prior to the start of the Sorter bout, the instructor inputs a list of Sorter subcategories that relate to the game's overall research topic.The Sorter bout randomly displays five sources and players must sort each source into the subcategory that best summarises the source's subject content.Players earn 1,000 points for each source up to quota and 1,500 points for each source thereafter.Quotas for the T&R and Sorter bouts vary based on the number of closed sources and the number of ratings per closed source.
BiblioBouts ends with the Best Bibliography bout.Players choose one of several instructor-formulated research questions and choose the best sources they would use in a written paper that addresses this question.For each source they choose, they earn 5,000 points up to the cap.The default Best Bibliography bout cap is ten sources.http://dx.doi.org/10.11645/5.2.1637 BiblioBouts' scoring algorithm awards increasingly greater numbers of points as players progress from initial to later bouts.It was thought that this approach would maintain students' interest in the game because they would be able to be competitive, earning more and more points as the game progressed.Additionally, the game gives players many opportunities to earn "bonus points."These points are awarded based on the extent to which their game-play activity matches the average activity of fellow game players.Because bonus points depend on the activity of class as a whole, the game computes them at the conclusion of each bout and adds them to students' scores.

Methods
Thirteen classes played BiblioBouts in the 2009-2010 academic year.Classes averaged 24 students.For this paper's analysis, we chose the two largest classes named "Introduction to Information Studies" (IIS) with 90 students, and "Video Games and Learning" (VGL) with 66 students.IIS students played BiblioBouts while writing a paper on the broad topic "Worklife quality" in which they were required to cite at least three publications from library databases.VGL students played BiblioBouts while writing a "game play reflection paper" in which they described their experiences playing educational games and answered the question, "How do people learn from video games?"When students play BiblioBouts, the game records their activity on game logs for subsequent analysis.Logs begin during the Donor bout when the game creates a source record for each source donated by players bearing a unique accession number, player identification number, date and time of donation, title, source, and URL.When players "close" sources, BiblioBouts adds a timestamp bearing the date and time of closing.BiblioBouts amends these records every time a player tags and rates the source, sorts it, or chooses it for their best bibliographies.These game-play data were analysed to determine levels of player activity and performance within the game.
Another source of data came directly from players themselves.Every time students played BiblioBouts, they could volunteer to complete an online game-diary form that collected information about their experience playing the game.At the end of the game, players were compensated with $5 per completed form.Also at the end of the game, the R&D team hosted focused group interviews with student game players.Students who participated in these focus groups were compensated with $25 cash and lunch.R&D team members also interviewed instructors before and after the BiblioBouts games were played in their classes to learn about their expectations for the game and students playing the game.These interview comments from both instructors and students helped researchers to better understand the analysis of game log data.
Together, the data gathered during and after game play was used to answer these five research questions:

1.
To what extent do players meet and/or exceed minimum-level game-play expectations for each of the game's individual bouts? 2.
How long do students play the game?Do they play continuously, in stop and starts, as deadlines approach, etc.? 3.
What are typical game-play styles?4.
What impact does the game's scoring system have on game play?http://dx.doi.org/10.11645/5.2.1637

5.
How can our knowledge about how students play BiblioBouts help to develop a model of best practices for the design, development, and deployment of information literacy games?
The R&D team's analysis of collected data also answered research questions about the skills and concepts BiblioBouts players learned as a result of playing the game; however, this paper is limited to the five research questions above.

Results
Quantitative game-play data gathered automatically by the game engine was analysed to understand player-behaviour statistics.These data give an overall view of the amount of time and effort players exerted.Qualitative responses provided by students in focus groups give more insight into individual player experiences.In combination, these two sets of data outline both how and why players performed as they did.This analysis helped the R&D team better understand player motivation and make improvements to the game in response.

Playing the Donor Bout
Although both instructors set this bout's quota at six sources, scoring encouraged students to donate more sources than the quota because it awarded them double the number of points ( 200) for each donated source above the quota.Small percentages (between 6% and 11%) of students did not play the Donor bout or failed to reach quota.This reflects players who simply refused to play or dropped out after the initial registration stages, which involved some technical problems.IIS players (17.8%) were more likely than VGL players (9.1%) to reach quota and stop playing the bout.In fact, the majority (52.2%) of IIS players were content to meet the donor quota or exceed it up to 4 sources.http://dx.doi.org/10.11645/5.2.1637 Although the group of IIS players was 1.3 times larger than the VGL group, this group contributed more (140) sources overall to their game than the VGL group contributed to theirs.Almost half (48.6%) of the VGL game's sources came from 10.5% of its players.Overall, VGL players averaged 18.8 sources.Subtract the donations of the three VGL "superplayers" who donated 77, 92, and 230 citations, and the VGL average for the Donor bout plunges to 13.5 which is comparable to the 12.2 overall IIS donation average for this bout.
Earning 100 points per donation up to quota, 200 points per donation above quota, and 20 points per unique donation, a total of three VGL superplayers earned 8,700, 9,000, and 24,000 points, building a towering lead over less prodigious players who merely doubled the Donor's quota (2,700 points).Right from the start, VGL students were "gaming the game." On average, IIS and VGL students played the Donor bout for 51.8 and 36.6 minutes and donated 12 and 18 sources, respectively.IIS students averaged 4.2 minutes to donate a source to the Donor bout.VGL students took only 2.0 minutes to accomplish the task.Such short average times were due to students' use of Zotero's "select all" search results capability that allowed them to save an entire page of retrievals in Zotero and submit them to BiblioBouts' Donor bout.For example, in an eight-minute period, a VGL player donated a total of 74 sources, all from Google Scholar, and earned a whopping 14,200 points.Because of the large number of titles donated per minute, it is doubtful this player scrutinized much more than each source's title to make sure they were on topic.There was no time to open each one, check author credentials, and read their abstracts to make preliminary relevance assessments.In focus groups, students confirmed our suspicions in this regard.
"It was easy to spam the system with nonsense and get lots of points for it, so I didn't even look at the sources.I submitted the same ones many times and it didn't matter." Students suggested changes to Donor bout scoring that would reduce prolific donors.
"It is bad that they give you more points for more sources-quantity over quality.It does not help the learning process at all.You should be able to pick a max of 15 sources." The IL goals of the Donor bout were to give players practice in selecting relevant databases, searching databases for sources on their topic, choosing appropriate sources, downloading full-texts, and verifying complete citations.Players were expected to use professional resource and discovery tools such as the library portal, scholarly databases, and Zotero.Both quantitative and qualitative data suggest that players did not devote appropriate time and effort to these skills, engaging instead in "gaming the game" and spamming donations to earn points.To reduce such game-play behaviour, we de-emphasized excessive donations in BiblioBouts" scoring algorithm (see the improvements section 6).

Playing the Closer Bout
Percentages of IIS and VGL students failing to meet the Closer bout's cap were low at 1.1% and 6.1%, respectively.Especially troubling were the percentages of IIS (22.2%) http://dx.doi.org/10.11645/5.2.1637 and VGL (33.3%) students who did not play this bout.In focus group interviews, students confirmed what the authors of this paper suspected, that is, students had difficulties finding full texts and using Zotero to save their sources.
Here is what students said in focused groups about their difficulties using Zotero.
"Zotero … it was too many steps to go through … We had to add the sources and go back and add the PDFs to the sources.If we could have just added in [everything] one step, it probably would have been easier, but I mean for me, it really does suck." "It was a little complicated to figure out how to attach the copy to this and but once you got it, all of the rest of the steps seemed easy to me.Which I guess in a perfect world, you just drag something over there and it'd be done but it takes a little more than that." The IL goals of the Closer bout were to give players practice in assessing relevance of search results, verifying and correcting citation and full-text attachments, using the Zotero citation management tool to manage sources.The technical challenges of using library databases, saving retrievals, and managing them with the Zotero plug-in negatively impacted the effectiveness of this bout.In section 6 we describe the ways in which we helped students tackle this bout's technical challenges.

Playing the Tagging & Rating (T&R) Bout
Table 3 describes the extent to which students played the T&R bout based on meeting and/or exceeding its quota.T&R quotas were different for the two classes because fewer sources overall were closed in the VGL game.In both classes, about one-third of students failed to play T&R or meet its quota.Of these, 46.4% of IIS and 56.5% of VGL students also failed to play Closer or meet its cap.When students played T&R, they usually exceeded its quota, tagging and rating another 4 to 9 sources.Five students were T&R superplayers, exceeding quota by two times or more.
Overall, IIS and VGL students took 3.4 and 2.6 minutes to tag and rate one source, respectively.Figure 1 shows that the more sources players tagged and rated, the quicker they were to finish the task.

Figure 1. Time spent tagging and rating a source
Students below quota took the longest, about 4.5 minutes per source.The shortest average times 2.6 and 1.9 minutes were registered by VGL players at quota or above.
The trend from longer to shorter time periods based on the number of tagged and rated sources may reflect a learning curve connected with tagging and rating tasks.That the times are so low, hardly two to three minutes, is alarming because evaluating a source should take longer than that.Sufficiently familiarising oneself with the paper's content in order to assess relevance means opening full-texts, reading abstracts, introductions, problem statements, and discussion sections.These tasks should take longer than the average times, plus there were more tagging & rating activities on top of relevance assessments for players to do.In focus groups, students told us they found shortcuts to reduce the effort they exerted on tagging and rating tasks."I don't think you have to necessarily read the whole thing.The abstracts and like, at least, introductions provided you an idea of the whole thing." The IL goals of the T&R bout were to give players practice evaluating sources.The analysis of game-play data suggests that large proportions of players from both classes failed to play or meet the bout's quota.When players met quota, few exceeded it at rates that characterized the Donor and Sorter bouts in which players tripled or quadrupled quota.In fact, these players were responding to the small scoring gains that were the reward for doing tagging and rating, tasks that required their attention and took time to complete earnestly.The improvements section (6) describes how we changed BiblioBouts' scoring algorithm, giving players greater rewards for reaching the T&R quota and encouraging them to exceed it.Students' participation in the game rebounded in Sorter because students earned many points for a small amount of effort.About one-third of the students who had been nonplayers or below-quota players up to the Sorter bout rejoined BiblioBouts during the Sorter bout to boost their scores.Students' observations on the small amount of effort they put into sorting, boosting their scores, and gaming the Sorter bout exposed systemic problems with this bout.

Playing the Sorter Bout
"I found a way to game the system with the sorting round where me and like four other people just threw 'em all in the top category and since we were sorting them in the exact same way, [our scores] jumped up a lot." " The IL goals of the Sorter bout were for students to become increasingly familiar with source content, and to organise sources by topic and subtopic in preparation for outlining and writing papers.The technical ease of performing this bout and the huge points gains for a minimal investment of time and effort were opportunities most players, even ones who had hardly played, found too good to miss.Because it is doubtful that students achieved this bout's IL goals, the R&D team redesigned BiblioBouts, eliminating the Sorter bout and distributing its tagging aspect into the T&R and Best Bibliography bouts immediately preceding and following it, respectively.Details of these changes are discussed in section 6.

Playing the Best Bibliography Bout
28.9% of IIS and 47.0% VGL students failed to play the Best Bibliography bout or meet its cap.One reason students failed to play this bout may be the insurmountable scoring lead established by the Sorter bout superplayers.Students made this observation in focus groups.
"And another thing is for people who are really far behind, I know one of my friends said that they were about 20,000 points behind the person ahead of them and they figured, "What's the point [of playing]?I'm not going to catch up." VGL's instructor made a comparable observation in a post-game interview: "The scores … started to diverge and some people started to pull ahead, that was motivating for the one group and demotivating for another group … People thought, "Well, I'm so far behind now, why bother?" Because everyone played Best Bibliography in one sitting, BiblioBouts time stamped this bout once, making it appear that players took one minute to complete this bout even though it might have taken longer for them to scrutinize citations, abstracts, and fulltexts.http://dx.doi.org/10.11645/5.2.1637 The IL goals of the Best Bibliography bout were for students to use relevance and credibility ratings to choose the best sources, and compile a vetted bibliography of sources linked to citations and full texts.While a large percentage of players did not finish the bout, those who did gained experience in judging and selecting sources on a narrow topic, and ended the game with a vetted bibliography.The many game improvements that are described in the improvements section 6 were made to ensure future game players engage in productive game-play behaviour from beginning to end so they get valuable experience conducting library research while they complete their writing assignments.

Scoring and Game-Play Styles
Game-play styles were based on the extent of student participation in the game across all five bouts.Table 6 lists and defines game-play styles of players in the two classes.

Partial-game play styles: In-game non-players and rejoiners
In-game rejoiner A player who plays beginning and ending bouts but does not play 1 or more middle bout(s) In-game non-player A player who fails to play two or more bouts

Non-player style
Non-player Student registers but plays no bouts Full-game styles were characterized by sustained-level game play from beginning to end.Partial-game styles that involved sustained-level game play included most drop-out styles because students played at a sustained level for a period of time before dropping out.Partial-game styles that did not involve sustained-level game play were in-game rejoiners and in-game non-players.Game play styles were distinctly different in the two classes.Over half (56.6%) of the students in the IIS class played BiblioBouts from start to finish.In the VGL class, a little under a quarter (24.3%) played BiblioBouts at sustained levels from start to finish.Realizing they were far behind the leaders' scores, a sizable percentage (27.2%) of VGL players dropped out after the Sorter bout.If VGL students had played at sustained levels instead of dropping out, the percentage (51.5%) of VGL students playing BiblioBouts from start to finish would have been comparable to the IIS class.
Most likely, IIS and VGL students played differently because they had different objectives.IIS students played the game in search of sources for writing a research paper.VGL students did the same but their papers focused on their experiences playing BiblioBouts including discovering the games' learning goals, motivational strategies, and positive and negative aspects.Thus, they were more inclined to "game" BiblioBouts, searching for flaws that would enable them to win or be amongst the top players on the leader board and reporting these flaws in their report to the instructor.
Especially troublesome were the over one-quarter of IIS and one-third of VGL students who were in-game non-players and rejoiners.Perhaps such inconsistent and unpredictable game play was due to the many distractions that compete for students' attention during the several weeks they are playing BiblioBouts.http://dx.doi.org/10.11645/5.2.1637 "I started off pretty competitive.I wanted to do well and my goal was just to be in the top three for the longest time.And then because the game takes so long and it's really such a large span of time, towards the end I didn't really care quite as much.Because there were other things going on in other classes and also in that class and so it kind of lost that for me." "Like don't drag [the game] on so long and make it more interesting for people to actually go on there and have fun while they're on there."

Improving the BiblioBouts Information Literacy Game
The two classes of undergraduate students that played the BiblioBouts game had different goals: (1) one class pursued the traditional goal of completing a research-andwriting assignment while playing the game and (2) the second class played BiblioBouts and other educational games to determine the potential learning or educational value of games into the classroom.Different goals resulted in marked differences between the two classes in terms of game play, which allowed the researchers greater insight into differing motivations and performance of different types of players and their level of engagement with each bout's IL skills practice.The VGL players' behaviour was most helpful to the R&D team in improving BiblioBouts game play, while the IIS players helped the R&D team predict the demands the game places on students and how the game is likely to benefit students.
Differences between game-play styles were revealing.The majority of IIS players enlisted full-game play styles but partial-game styles characterised most VGL players.
Realising that the insurmountable lead some students were building was causing other students to drop out, the R&D team traced scoring problems to the Donor and Sorter bouts that had no limits on the number of donations players made or the number of sources they sorted.Sorter was especially problematic because the effort players made to sort sources was inversely proportional to the points the game awarded.
Before the R&D team could evaluate the learning gains players made as a result of playing BiblioBouts, improvements had to be made to BiblioBouts so students engaged in productive game-play behavior and played the game in earnest from beginning to end.
Here are the major improvements that the R&D team made to BiblioBouts.
To discourage Donor spamming, the R&D team eliminated bonus points to players who exceeded Donor's quota by 1.5 times.
On the surface, it appeared that reducing bonus points for Sorter bout activity beyond the quota would "fix" this bout.Unfortunately, Sorter had other problems.For example, devoting an entire bout to sorting put undue attention on source categorisation, and the game's failure to use Sorter categorisation results in subsequent bouts forced the R&D team to rethink this bout's value.As a result, the R&D team eliminated Sorter bout altogether from BiblioBouts and transformed its categories aspect into keywordgeneration tasks for the T&R and Best Bibliography bouts.In a redesigned T&R bout, players tagged the sources they evaluated with keywords, and they earned bonus points when their keywords matched those of other players performing the same task.BiblioBouts also enhanced source records with matching keywords and displayed them in the Best Bibliography bout.In a redesigned Best Bibliography bout, players formulated http://dx.doi.org/10.11645/5.2.1637 a topic for their writing assignments based on the broad-based topic in play and entered three keywords that described the themes that their papers would address.BiblioBouts rewarded players whose best bibliography sources matched the keywords players had added to sources in the T&R bout.Minus the Sorter bout, BiblioBouts game play would be reduced by a week, and the keyword-generation task now had greater utility in the game.
To build a new scoring algorithm for BiblioBouts, the R&D team programmed an Excel spreadsheet that modeled the game-play styles of quota, above-average, and superplayers.We used the spreadsheet to ensure that players whose game play was in keeping with the following four scoring priorities would earn a spot high atop the leader board: (1) meeting the game's caps, (2) exceeding its quotas, (3) agreeing with their opponents' credibility and relevance ratings and content tags, and (4) being the first to close the sources that their opponents choose for their Best Bibliography.Especially challenging has been scoring the expanded T&R bout so that it rewards players who exceed quota but does not allow them to open up a lead that other players feel is insurmountable.On one hand, we want players to exceed the T&R bout's quota because players receive more practice evaluating sources and more exposure to sources on their paper's topic.On the other hand, we want players who close the best sources that most opponents choose for their best bibliography to place high on the final leader board.A new scoring algorithm is now in place.Refining BiblioBouts' scoring algorithm will be an on-going task that occupies the R&D team.
To train students in the use of technology of the library-research process, the R&D team strongly encourages instructors to invite librarians to classes before students start playing BiblioBouts.Librarians demonstrate their institution's database portal, relevant databases for the topic in play, and using Zotero to save and manage retrieved sources, citations, and full-texts.Game invitations that BiblioBouts sends to players via email at the beginning of the game include suggested library databases and keywords to get students started finding sources.BiblioBouts "help" includes cheat-sheets describing how to use Zotero especially for saving full-texts.
The large percentages of non-player and rejoiner styles was also another problem that the team had to address.These styles were characteristic of students who lost interest in the game, the latter rejoining the game to take advantage of game play such as Sorter that gave them many points for quick and simple tasks.In focus groups, students told us they wanted to have fun.
"Make it fun.At the beginning, draw me in.Get the hook." "I don't really know if there could be like a fun factor added … I don't even know what could make it more fun to be honest.It's like a research … There's really not a fun aspect to that.But if there could be."Some players were doubtful BiblioBouts could ever be fun because they would be graded on their participation in the game.
"Because essentially it is going to be graded … The BiblioBouts game itself, it's a part of our grade so that's why I saw it as an assignment.And like the game itself like finding sources, it was helpful definitely but it was another assignment."http://dx.doi.org/10.11645/5.2.1637 As a result of this feedback, the R&D team has added these game-like features to the beta 2.0 version of BiblioBouts that debuted in fall 2011: • Badges for satisfactory game play.For example, players earn Roadrunner Badges for being the first to close or rate and tag, Speed Demon Badges for being the first to reach the bouts' quotas or caps, and Best Collection Badges for being the ones whose sources are cited most frequently in their opponents' best bibliographies.
• Levels based on points earned.
• A recent-actions report that describes what the player did and how many points were awarded.
• A detailed scoring log that describes all player actions and points earned from the start of the game to the present moment.
• A personal trophy case that displays a player's level, badges, and current score.
• A public trophy case accompanying source evaluations that displays the rater's current level, badges, and leader board standing so that players can decide how seriously they want to consider the rater's evaluation.
The goal of these changes was to increase BiblioBouts' game-like functionality and make BiblioBouts more fun for students to play.Data on future game sessions will be analysed to establish whether these improvements lead to fewer dropouts, more fullgame players in the future, and thorough game play that yields relevant and credible sources on the broad-based topic in play for all students to use in their papers and thoughtful comments on source content, relevance, and credibility.

Discussion
BiblioBouts is unique because it fosters students' IL practices within a gaming scenario as they play this online information literacy game while they complete a writing assignment.It puts professional search tools in their hands, gives them practice utilising a methodology for evaluating sources on a topic, and gives them opportunities to find relevant sources from the collection of all sources that they and their classmates find on the broad-based topic in play.
As the paper has shown the R&D team's initial conception of BiblioBouts was not without flaws.Its scoring algorithm that awarded increasingly greater numbers of points as players progressed from initial to later bouts was especially problematic.
The R&D team conducted an in-depth analysis of game-play data to determine how and why students played and failed to play the game.Data collection was multi-modal combining quantitative methods such as surveys and game play logs with qualitative methods such as focus group interviews, personal interviews, and online diary forms so that we could utilise qualitative data to explain the quantitative data in game-play logs.
As a result of the analysis, the R&D team redesigned BiblioBouts, eliminating features of bouts that failed to achieve IL goals, redistributing features across several bouts to http://dx.doi.org/10.11645/5.2.1637 achieve greater flow and cohesiveness and adding more game-like features and improving the game's scoring algorithm to increase student interest and engagement.Despite the game's flaws, students in focus group interviews extolled the game's benefits:

•
Realizing that library databases yield sources that are qualitatively better than Google, Wikipedia, and the web • Getting hands-on practice using a step-by-step approach for conducting library research and evaluating sources • Finding relevant sources for their writing assignments that other students submitted to the game • Reducing procrastination • Giving access to many times more sources than they would have found on their own Now that we have an improved game especially one that is bereft of the loopholes that enabled students to effortlessly earn big-time points, the R&D team is poised to undertake a comprehensive evaluation of the learning gains BiblioBouts players receive as a result of playing the game.Had we done so earlier, results would have been marred by the game's design inefficiencies and shortcomings.
Building a web-based information literacy game has been a complicated process requiring instructors with a pioneering spirit who will welcome a work-in-progress into their classes, exhibit patience and understanding while unexpected technical problems are overcome, and participate fully in the evaluation and improvement of the game.
Students especially are indispensable for evaluating the game.We advise IL game developers and IL librarians assisting them to take a cue from our game-design experience.Game development and evaluation go hand in hand when the objective of the evaluation is to build a game that players play in the ways that the game designers intended the game to be played.Once that objective is achieved, the evaluation can focus on what players learn as a result of playing the game.
Findings also inform instructors who are considering adding BiblioBouts to their classes and want to know the demands the game places on students.Although the R&D team's game-redesign efforts reduced the game's bouts, it produced a streamlined game that focuses players' attention on vetting the sources they and their classmates contribute to the sources database on the broad-based topic in play that everyone can use to complete a research-and-writing assignment.

Summary
The impetus for using games to teach students information literacy skills and concepts was the R&D team's recognition that games could usher students through the libraryresearch process, giving them hands-on practice while they completed a writing assignment and benefiting everyone by pooling their information-gathering efforts into a shared database of all the sources students find on the broad-based topic in play.The http://dx.doi.org/10.11645/5.2.1637 R&D team designed, developed, and evaluated the BiblioBouts information literacy game.Although the evaluation yielded useful data about the game's learning gains, the team immediately put to work evaluation findings to improve the game and ensure that future game players will engage in productive game-play behaviour from beginning to end.In the future, the R&D team will analyse game-play data to determine whether game-system improvements produces the desired behaviour as well as leads learning gains.Librarians who are either contemplating games to students IL skills and concepts or teaming with game design and development teams are advised to put their interest in demonstrating learning gains aside until they are certain that the design of their games achieves the desired game-play behaviors.

Table 1
describes the game's five bouts and enumerates the IL skills, concepts, and tools that students learn and practice during game play.

Table 4 . Sorter Player Types Based on Meeting Quotas
Game-play activity rebounded in the Sorter bout with less than 20% of students failing to play.Above-quota play was the norm with 61.0%IIS students and 75.8% of VGL students exceeding the Sorter quota.Sizable percentages (17.4% IIS and 15.2% VGL) sorted over 200 sources.Table4describes the extent to which students played the Sorter bout based on meeting and/or exceeding its quota.Because BiblioBouts presents students with 5 sources at a time, most sorted above quota 7 sources, IIS and VGL students averaged 18.2 and 24.7 minutes to play the bout.An average Sorter player in the IIS and VGL games earned 56,000 and 87,500 for less than a half hour's work.http://dx.doi.org/10.11645/5.2.1637 [Sorter]scoring … didn't really like have anything to do with like how much effort you put into it … You could like pretty easily just like click stuff and get a really high score … [There were] ways that you could like increase your score like by not really doing too much work … You could … drag 'em wherever you wanted or just drag 'em … really quickly and that one was worth a lot of points … That basically made the first few rounds like not matter at all."

Table 6 . Game-play styles and definitions Styles Description Full-game play styles
QuotaPlayer meets or modestly exceeds all bouts' caps and quotas Above quota Player meets all caps and modestly exceeds two or more bouts' quotas (by about 10 Donor or 10 T&R sources or 20 Sorter sources) Superplayer Player meets all caps and more than doubles two or more bouts' quotas Sorter superplayerPlayer is a quota player except for Sorter which s/he more than doubles its quota

Table 7 . Game Play Styles of IIS and VGL Players
Table 7 lists the number and percentages of IIS and VGL players per style. http://dx.doi.org/10.11645/5.2.1637