The Amazing Library Race: tracking student engagement and learning comprehension in library orientations

Seeking to introduce first-year students to library resources and services in an engaging way, an orientation titled The Amazing Library Race (ALR) was developed and implemented at a university library. Informed by the pedagogy of problem-based learning, the ALR asks students to complete challenges regarding different departments and services. This study assesses this initiative’s success using observational and artifact-based data, addressing the challenging prospect of evaluating the impact of library orientation sessions. Two rubrics were developed to measure student involvement and student learning comprehension. More than 14 hours of in-class observations were used to track engagement, and 64 artifacts of student learning were collected and coded to evaluate learning comprehension. After coding, interrater reliability was established using the intraclass correlation coefficient to establish the validity of the ratings. This paper will outline these methodologies, present the results of the data analysis, and discuss the possibilities and difficulties of measuring student engagement in information literacy instruction centred upon active learning.


Introduction
As academic libraries strive to improve student experiences many institutions have sought to establish positive relationships with first-year students to increase academic performance, graduation rates and retention.The library instruction programme at Long Island University Brooklyn strives to actualise active and student-centred learning opportunities, and towards this end the assessment of a redesigned library orientation was undertaken.An instructional initiative titled 'The Amazing Library Race' (ALR) was evaluated in order to better understand the levels of student engagement with their peers, library faculty and staff, and library resources.Library inductions and orientations, frequently functioning as introductions to library services and resources for first-year students, often act as an important precursor to information literacy (IL) instruction.This is the case with the Amazing Library Race, which orients students to key aspects of the library while simultaneously setting the stage for future IL instruction also rooted in active learning.With an understanding of the library's physical space and major service points obtained from the library orientation, students are better positioned to grapple with more complicated concepts related to information and libraries in subsequent classes and coursework.
The Amazing Library Race is based on the reality TV game show The Amazing Race, wherein contestants travel the world to undertake various challenges and compete for a cash prize.The ALR was first developed by the University of Arizona (2006) to introduce first-year students to the libraries' staff, collections, and facilities, and their success inspired other institutions to adopt the model.At Long Island University the ALR has been modified to suit the estimated needs, backgrounds, and skill levels of our diverse student body.In particular, the instruction has been adapted to fit the unique requirements of the first-year orientation programme the students are enrolled in, which is intended to accustom students to college-level academic expectations and university life.
The Amazing Library Race follows a similar process for each session.Students assemble into teams of three to five people, and each group is given an answer sheet to complete as well as a card containing the challenges of the first leg of the race.The tasks address common resources and services that students are likely to utilise during their coursework, but with a decidedly problem-based approach to such content frequently addressed in orientations.The challenges involve activities such as participants finding a book in the stacks and drawing a picture of its cover, writing a haiku about specific library services, recreating a DVD cover from the Media Center by dressing in props and photographing themselves, and locating answers to trivia questions in reference books.A total of 185 students in twelve classes participated in the ALR in autumn 2013, while autumn 2014 saw the involvement of 227 students in fifteen courses.Following the piloting of the programme, the authors sought to assess this initiative's success in terms of student engagement and comprehension.A review of the literature will provide useful context prior to describing the study's methodology for measuring these factors.

Approaches to library orientations
Many college and university libraries have designed orientations that introduce students to the library in an informative and approachable manner.To determine the efficacy of two instructional approaches Marcus and Beck (2003) compared a scavenger hunt orientation to a traditional tour, concluding that the active learning approach resulted in higher attainment of learning objectives and a more positive student attitude regarding the experience.Of the academic librarians creating active learning opportunities some chose to incorporate formats mirroring those found in popular culture and on television into their instruction, such as Walker's (2008) use of Jeopardy!-themed class sessions and Springer and Yelinek's (2011) adoption of selected television clips to create personal connections to the material being taught.Popular culture examples have also been found to be useful for illustrating information literacy concepts and designing authentic activities (Bach 2011;Tewell 2014).One such example is that of the The Amazing Race, upon which several libraries have adapted library orientations (Xavier University 2013).Eckenrode (2008) reported on the use of the Amazing Library Race at the State University of New York at Fredonia, including an appendix containing lesson materials, and librarians at Southeastern University wrote on their adoption of the problem-based orientation to increase student engagement (Banks and Svencionyte 2008).

Student engagement and problem-based learning
Problem-based learning (PBL) is a pedagogical method wherein participants gain knowledge or skills by working in groups to explore and respond to a specific question, problem, or challenge.Originating in McMaster University's school of medicine in the 1960s, PBL replaces traditional lecture-based instruction with active and collaborative learning.Barrows (1996) outlines the primary components of PBL, including student-centred learning, the formation of small groups, teachers as guides, and the assignment of a problem as a learning stimulus.In the PBL classroom teachers intend to act as facilitators rather than authorities, and provide encouragement for students instead of dispensing information.A number of library and information science (LIS) researchers have detailed the benefits of PBL for increasing student engagement (Snavely 2004;Spence 2004) as well as its potential application to stand-alone one-shot sessions (Kenney 2008).Experimental studies in library settings comparing two methods of instruction have shown PBL to improve the attainment of learning outcomes in large workshops for first-year students when compared to lectures (Hsieh and Knight 2008) and to increase engagement with library resources through course-integrated instruction sessions (Hines and Hines 2012).

Assessment of information literacy instruction
Many methods of assessment for IL and student learning have been presented in the professional literature.Oakleaf (2008) and Sobel and Wolf (2011) offer useful frameworks and approaches to assessment, describing the applicability and efficacy of various tools according to one's purpose in conducting assessment.Specific classroom assessment practices used to guide IL instruction can range from pre-and post-tests coupled with citation analysis (Gilbert 2009) to evaluating electronic portfolios using rubrics (Diller and Phelps 2008).While the evaluation of learning outcomes in various forms of library instruction has been well documented, very little research exists regarding the assessment of library orientations specifically.Notably, Brown et al. measured library anxiety among first-year students before and after instruction and found decreased levels of library anxiety post-orientation (2004, p. 398).Of the many options for assessment in the library classroom, rubrics have been suggested as a versatile and locally-adaptable tool, and one that avoids some of the numerous drawbacks of test-based assessments.
Rubric-based methods of evaluation have been advocated for and adopted by numerous LIS researchers as an approach to assessing student learning, particularly when accompanied by a rigorous norming process (Fagerheim and Shrode 2009;Oakleaf 2009;Knight 2006).Though requiring significant time and practice to properly norm and validate, rubrics can facilitate the evaluation of a wide range of IL instruction scenarios, from single sessions to semester-long credit courses (Holmes and Oakleaf 2013).One major venture devoted to exploring and encouraging the use of rubrics for IL, titled Rubric Assessment of Information Literacy Skills (RAILS), is a grantfunded research project providing access to a number of rubrics developed by academic libraries in the United States (RAILS 2014).With these assessment measures and tools having been reviewed the authors developed a methodology for the study at hand, with the goal of evaluating elements of engagement and learning comprehension exhibited by students participating in the ALR.

Methodology
Rubrics were chosen as the method of assessment of the ALR to ensure higher objectivity in grading and to lend a measure of validity and reliability to the findings on student engagement and group learning comprehension.Two separate rubrics were developed, normed, and applied to the ALR; the first was used to standardise observations of student engagement, and the second to improve grading consistency of the student answer sheets.A draft of each rubric was collaboratively developed among the raters, and two rounds of norming were conducted to ensure a more unified application of each rubric.During these sessions, each rater coded a sample set of artifacts, or in the case of the observation rubric, each rater completed a sample observation session.The raters then met to discuss and reconcile differences in their coding, and once a degree of consensus was reached, each rubric was edited to reflect these changes.This process of norming is crucial to ensuring that all raters understand and apply the rubric in a similar manner (Reddy 2011) and thus reach an acceptable level of interrater reliability (IRR).The final observation rubric (see Table 1) was used during class to assess student-to-student and student- Boss, Angell & Tewell. 2015. Journal of Information Literacy, 9(1).http://dx.doi.org/10.11645/9.1.1885 7 to-library-faculty engagement.The group learning comprehension rubric (see Table 2) was used to code the students' answer sheets.Left question blank or incorrect answer.
Incomplete answer.Answer selected matches fact in book.
Each of the three researchers independently graded all 64 student answer sheets with the learning comprehension rubric.IRR was then calculated on the statistical computer program SPSS using the intraclass correlation coefficient (ICC) to ascertain the degree of agreement among all three raters.ICC is frequently used for projects encompassing more than two raters (Morgan et al. 2012).The ICC model selected was two-way random and the type was absolute agreement with a confidence interval of 95%.IRR was calculated using a standard reliability analysis and basic descriptive statistics were used to assess data gathered from the observation rubric.

Results
The analysis revealed that high IRR levels were achieved on all seven criteria of the learning comprehension rubric (see  The observation rubric was used to assess in-class measures on three different indicators: student-to-student engagement, student-to-library-faculty engagement and workshop duration.Descriptive statistics were employed to measure these three different criteria, which is displayed in Table 4. Overall, students scored very well on all three rubric criteria.The best performance was displayed on the workshop duration indicator, with an average of 1.9 out of 2 (exemplary).Student-to-student and student-to-library-faculty engagement were nearly identical, with the former prevailing by a slight 0.2.The lowest score was student to faculty engagement, an expected occurrence in an activity which separates students into teams.
In order to directly measure learning outcomes the learning comprehension rubric was used to grade the answer sheet.Group learning comprehension was assessed as above proficient for each of the seven tasks (see Table 5) in the ALR.The highest average comprehension was found for Question 5, which requires students to find a book in the stacks using the call number, and then draw a picture of what they think the book cover should look like based on its title.On a scale where 0 is below proficient, 1 is proficient and 2 is above proficient, average learning comprehension for this question was measured at 1.89.The lowest average learning comprehension was found for Question 6, which tasked students to write a haiku about how to print documents from the reference computer stations.Average learning comprehension for this question was measured at 1.52, still significantly above proficient.None of the answer sheets yielded below proficient scores.

Discussion
The IRR calculations concluded that all three researchers were in high agreement with regard to protocol for grading student artifacts with the learning comprehension rubric.This is a significant finding because it demonstrates the reliability of the rubric across both raters and time.Instead of assuming that all researchers are consistent in their grading practices, the IRR analysis offers statistical proof of objective assessment methods.In other words, the level of bias which typically increases with additional graders in an assessment situation has the power to be controlled and lessened by IRR analyses.High IRR demarcates a particular rubric as a valuable and dependable assessment tool, in this case signifying that all three researchers were interpreting the tool in the same manner to evaluate student performance on seven library-related tasks.The learning comprehension rubric can thus be applied with confidence to future ALR artifact assessment and potentially extended to other librarians interested in measuring similar dimensions.The high learning comprehension averages determined by rubric analysis were not unexpected, given that the ALR is designed to discourage failure: student groups must complete each task at the most basic level in order to advance to the next leg of the race.However, using the rubric to reliably score each answer sheet allows for a much finer granularity in the analysis of learning comprehension.Time constraints during the race allowed the librarians to make only the most cursory assessment of each task; there was often not an opportunity to check for more than whether the task was completed and the most basic elements answered or included.Coding allowed for a much more thorough evaluation of each group's responses, and revealed a range of proficiencies on each task.For example, the third task in the race asked students to correctly identify the different components of a provided Modern Language Association (MLA) style citation.Time pressures during the race allow the teaching librarian to make a quick assessment of the citation components in order for the group to proceed to the next leg of the race.Closer examination during coding revealed the level of understanding for each student group on the differentiation between journal titles and article titles, and many common errors were found.However, students still averaged well above proficient on even the most difficult tasks.The results of this analysis suggest the workshop was successful in its aims of empowering students to collaboratively gain basic knowledge of library resources and services.
Given that the ALR session is the first time students in this study visited the library for information literacy instruction (ILI) it is quite encouraging that they scored so highly on the seven tasks on the assignment.The fact that they completed the activity in groups and could pool their unique knowledge and experiences into solving the problems is proposed as the primary reason for these notable scores.Empirical evidence to support this hypothesis does not exist, as every ALR participant worked in a group.An interesting future experiment could require students to complete the tasks alone and then compare their individual scores to the marks of the students who carried out the activity in groups.

Limitations and future directions
Throughout two years of administering and assessing the ALR several limits and challenges relevant to the goals and implementation of the activity surfaced.First, although many students and faculty have voiced their support of the ALR, others indubitably prefer a traditional, lecturebased format.A multitude of learning styles exists, and the ALR's problem-based group method is not the preferred style of everyone.The increasing popularity of active learning methods coincides with empirical studies reporting the enduring value of lecture-based instruction.In 2011 Covill surveyed 51 undergraduates on their perceptions of lectures within the psychology classroom.She discovered that the participants overwhelmingly considered lectures an engaging, long-lasting and 'excellent' pedagogical method.In contrast, some students appeared wary of active learning, claiming that they already receive benefits typically associated with active learning, such as engagement and independent thinking, from the professors' lectures.
The idea that an instructional switch from passive to active learning would pique confusion and apprehension among students is not overly startling, as for the majority of modern higher education history lectures have served as the pedagogical norm.The findings of Covill and others exploring these topics is of great value to proponents of active learning, as it serves as a reminder that these teaching methods are new to many students and should be applied with acknowledgment of and respect for their relative unconventionality.In terms of the ALR, each instructor provides students with an explanation of the activity at the beginning of the class, as well as clearly written clue cards for each task.Additionally, all students are required to return to the library for ILI at least two additional times during their undergraduate years.All of these classes contain a research component which warrants at least some of the class devoted to a lecture from the librarian.
In addition to these challenges, a limitation of the ALR involves the reliability of data generated by the observation rubric.Although the instruction department at Long Island University currently lacks the personnel to assign two librarians to each Orientation Seminar 1 library instruction, a team-teaching approach would allow for two raters to assess the student-to-student and studentto-library-faculty engagement.The benefits of multiple raters within social science research are many; advantages include increased accuracy and fairness as well as less introductions of bias into the results (Kane et al. 2013, p. 276).Therefore, the data generated from the observational rubric is less reliable than it would be if librarians administered the ALR in pairs.
There are presently several future plans for the continued growth and development of the ALR.
The ALR first debuted as a limited, pilot project.However, the combination of ALR requests by teaching faculty and the promising assessment results from the pilot led to an expansion of the workshop among nearly all Orientation Seminar 1 sections, a notable increase since the programme's inception.In the upcoming semester the researchers hope to expand the teaching roster even further, and explore the possibility of utilising graduate students to co-teach with a librarian.This would help to strengthen observational assessment data by incorporating a measure of interrater reliability.
This study also presents opportunities for further research on assessment of PBL orientations and instruction.Further direct assessments of student learning outcomes in library orientations are needed.Among the few assessments of library orientations that have been done, most do not go beyond measures of student satisfaction.While such indirect measures are valuable, they should be supplemented by direct assessments of learning comprehension in order to sufficiently evaluate the effectiveness of problem-based learning instruction.The ALR directly assessed student engagement and learning comprehension at the group level, so as to measure peer collaboration.However, additional assessment of engagement and learning comprehension at the individual, student level would make a valuable addition to the literature.Such research would allow for the possibility of further longitudinal studies related to student engagement with their peers, the library and its faculty and staff, as well as student retention and information literacy skills.

Conclusion
Assessments indicate that the ALR was successful by several measures.Nearly all ALR groups completed the activity within the allotted time frame, suggesting that despite a lack of any previous research instruction, the tasks were not too difficult for a first library visit.The fact that all students had at least two teammates to help them played no small role in their successful completion of the activities.Observations by all three researchers yielded reports of intensive student collaboration; students failing to participate were very much an exception rather than the norm.Data generated by the observation rubric further indicates the ALR is capable of sparking high levels of student-tostudent and student-to-instructor engagement.These are two factors valuable in any classroom, but it is especially important to cultivate them among students new to both academic libraries and higher education in general.
By providing students with problem-based learning tasks and a collaborative space in which to solve them, the ALR succeeded in encouraging students to reach out both to each other and the library faculty.These interactions can provide a solid foundation for first-year students to seek research assistance from librarians in the future.In addition to providing an engaging environment in which to learn, the ALR also resulted in high levels of student group learning comprehension of basic research tasks, such as reading a call number, finding a book in the stacks, accessing reference materials, and recognising components of a scholarly citation.Average group learning comprehension on all tasks was shown to be significantly above proficient, an indication that the ALR is more than just engaging, it is also a valuable learning opportunity.These findings will be used to make the case for the continuation of the ALR.This study's implications for the IL community include progress made towards assessing the efficacy of problem-based instruction in the academic library setting, particularly the challenging task of evaluating library orientation sessions.The assessment process described, including rubric development and validation and the collection of observational and artifact-based data, is one of the first direct assessments of library orientations found in the literature.It presents an alternative to more commonly used pre-and post-tests designed to measure learning comprehension or effectiveness of library orientations.Such fixed-choice instruments, while valuable, offer only indirect assessments of student learning.The assessments outlined in this study offer a more authentic measure of student abilities and behaviours, and can be easily adapted by librarians seeking to appraise similar modes of instruction at their own institutions, including at school and public libraries.
While many previous studies have used either anecdotal or qualitative assessment techniques the ALR is unique in that it applies both observational (qualitative) and rubric-based (quantitative) methods.Another implication of note is that IL assessment focuses almost exclusively on evaluating the work of a single student, but the methods developed for the ALR support assessment at the group level.This example of successful assessment beyond the individual level is a positive development, particularly in light of pedagogical shifts towards in-classroom collaboration.Most importantly, the results indicate that developing engaging IL sessions for undergraduates and effectively assessing this same instruction is an attainable goal.As information literacy teachers it is imperative that we continue to explore and evaluate the efficacy of our instructional methods with the aspiration of providing the best possible opportunities and environments for student learning.

Table 2 : Group learning comprehension rubric
Boss, Angell & Tewell.2015.Journal of Information Literacy, 9(1).http://dx.doi.org/10.11645/9.1.18858 (LeBreton and Senter 2008)generally considered the minimum acceptable level of IRR(LeBreton and Senter 2008), with anything below it signifying a problematic lack of consistency among raters.The highest IRR, .969,was for Q1 (checking the library catalogue for a book by a specific author) and the lowest IRR, .865,was on Q4 (completing a Mad Lib about textbooks on reserve).