Evidence-based instruction: assessing student work using rubrics and citation analysis to inform instructional design

This paper presents the findings of a study carried out by librarians in Champlain College who developed a two-pronged authentic assessment approach to measure the information literacy (IL) levels and determine the information seeking habits of students while conducting research for academic purposes. Librarians devised and developed an IL rubric and a citation analysis checklist for the assessment of first-year annotated bibliography assignment papers. This paper illustrates the merits of rubric-based, citation analysis assessment measures using authentic student coursework as a highly effective method of determining student outcomes assessment and information seeking habits while engaging in academic research. Findings from this study also suggest that authentic assessment is an extremely useful tool for instruction librarians to identify areas of IL that require further instructional support. This study is of importance to librarians wishing to adopt rubric-based and citation analysis authentic methods for student outcomes assessment. This paper is based on a presentation of the same name delivered at LILAC 2014.


Introduction
In academic year 2013-2014, two librarians in Champlain College (authors of this paper) undertook an intensive, deep assessment project using authentic examples of student work completed as part of their general education coursework requirements.Sampling first-year student coursework annotated bibliographies, we engaged in rubric-based assessment for evidence of information literacy (IL) competency.Following this rubric-based assessment, we also examined each citation from the annotated bibliographies.Using a specially designed citation analysis checklist, we studied over 500 citations, employing a citation analysis checklist to help us understand how students utilised library collections and information resources when faced with a real-life assignment.This paper outlines the findings of this study, which our instruction librarians later used to restructure and redesign instructional efforts in response to student needs.

COR 120 Annotated Bibliography
During the spring semester of their first year, all Champlain College students study the 'COR 120 Concepts of the Community' Core course.This interdisciplinary course is designed to have students explore the ideals, values, structures and traditions of communities as part of their liberal arts, general education.All students are required to complete a common assignment as part of their studies.This common assignment is comprised of a preliminary annotated bibliography based on a non-fictional community group, investigated from a social, political, economic, religious or cultural perspective.Students often choose to conduct research on historical communities such as Vikings, Puritans and Romans.More recently, we have seen students research community types from a less academic or scholarly, popular culture background.IL instruction is embedded within the COR 120 class.This instruction session supports students as they prepare to complete this common research assignment.

Assessment of information literacy
Much has been published recently on IL outcomes assessment.Many approaches to measuring student IL have been adopted, from standardised tests, rubric-based assessment and other methods.
Outcomes assessment has many purposes.Ultimately, Biggs and Tang (2011) suggest that assessment "matches performance as it is, with performance as it should be" ( p. 196).In addition, assessment is an important tool in helping librarians design and deliver instruction in a systematic and accountable way.Oakleaf (2009a) describes an IL instruction assessment cycle aimed to bring about institutional discourse on instructional quality and accountability.Following a seven-step process, this cycle begins by reviewing learning goals, setting learning outcomes, creating learning activities, teaching to learning outcomes, gathering data to check learning, interpret that data, and enact decisions to close the loop and make changes based on findings.According to Oakleaf, IL assessment enables practitioners to demonstrate how IL instruction contributes to student learning and development (2009a, p. 549).In a review of the literature on assessment and information

Rubric-based assessment
Rubric-based assessment has gained much recognition and credence in IL assessment in recent years.Described as "a set of criteria for students' work that includes descriptions of levels of performance quality on the criteria" (Brookhart 2013, p. 4), the benefits of using rubric-based assessment have been well documented:  Helps students learning more effectively  Students understand the expectations of instructors  Grades become more meaningful  Makes explicit what students are expected to learn  Facilitates self-evaluation  Promotes deep learning From the librarian's perspective, there are also benefits to using a rubric: it helps librarians agree upon learning expectations and outcomes, provides rich assessment data, and helps improve instruction.
Despite the benefits, there are also some drawbacks to the use of rubrics in assessment.When poorly written, the use of rubrics can be difficult and unproductive.Devising and assessing using rubrics is also a time-consuming effort, often requiring practice and adjustment (Oakleaf 2009b).Issues of inter-rater and intra-rater reliability are often associated with the use of rubrics, and special attention is needed to norm the rubric, as well as test for these issues (Holmes and Oakleaf, 2013).

Authentic assessment
Authentic assessment refers to a learner-centred model of evaluation, where a 'holistic' impression of assessment in gathered using authentic student work.Booth et al (2014, p. 2) refer to authentic assessment as an evaluation of learning experiences from the 'real world'.Biggs and Tang (2011, p. 212), argue that assessment must be based on "performance of understanding … that requires an active demonstration of the knowledge in question".Indeed, following an authentic assessment model, students are asked to complete "real-life tasks" that demonstrate performance outcomes (Suskie 2009, p. 26).As Biggs and Tang (2011, p. 213) conclude, for authentic assessment to be considered authentic, it must be an assessment of the total performance, and not just aspects of it.Authentic assessment is considered the ideal assessment approach for showing evidence of holistic, performance-based, campus-integrated student performance (Knight 2006, Oakleaf 2010).Despite this, using examples of student work for holistic assessment of IL is not commonplace (Diller and Phelps 2008).
Other studies have documented the use of annotated bibliographies as such an authentic assessment artefact (Rinto 2013).Devising a specific IL rubric that assessed five criteria: currency, relevance, accuracy of information, and authority and purpose, the author assessed a sample of authentic student annotated bibliographies for evidence of source evaluation.The author limited this effort to the assessment of this one single IL outcome, rather than multiple outcomes.In a similar rubric-based, authentic assessment effort, Hoffmann and LaBonte (2012) determined that the annotated bibliography assignment was not only a reliable measure of IL levels of students, but the assignment also provided insight into the research process undertaken by students due to its reflective nature.According to the authors, students used the annotated bibliography assignment to reflect upon their searches, and demonstrate evidence of research strategy revision, which was not as easily ascertained from final written papers.According to Hoffman and LaBonte, assessing annotated bibliographies provided greater insight into student performance, as well as the cognitive processes students undertake when engaging in academic research..

Instructional redesign
Formative and summative assessment efforts are an important stage in the instructional design cycle (Booth, 2011;Mullins, 2014).According to Mullins (2014), summative assessments in particular, allows for instructional modifications to guide IL efforts at a programmatic level.Formative assessment, on the other hand, allows for instructional modifications at the session level.In Mullins' IDEA model for instruction design, assessment forms the final stages in efforts to assess effectiveness of instruction interventions (Mullins 2014).Similarly, in Booth's USER model of reflective library instructional design, assessment forms a part of the reflect stage, which librarians use to assess the impact of instruction and revise the approach based on the findings (Booth 2011).Assessment forms an important part of the instruction design approach, and is a continuous, iterative process.

Citation analysis and collections
Citation analysis has played an important role in information science since Eugene Garfield's original studies on impact factor more than four decades ago.Citation analysis came to prominence in the sciences, where it was first used to measure the impact of authors through the proliferation of citations to their work (Garfield 1972).
Since the inception of this practice, librarians have used this means of analysis for a variety of ends, with the findings primarily used to guide collection development.However, less fully explored is the use of student bibliographies or citation lists to understand students' use of library collections for academic purposes.In such cases, researchers have placed their focus on journal citations.Typically, it is the bibliographies of theses or dissertations of graduate students that have been mined for data, and the majority of this data comes from the sciences and social sciences.Conkling et al. (2009) investigated 55,000 citations of doctoral students in ten subject areas (in the sciences and social sciences) in 'pre-web' and 'post-web' periods.They uncovered a shift to greater reliance on journal resources and conference proceedings in the progression from pre-to post-web and a concurrent decline in monograph usage.Sylvia (1998) investigated 157 psychology undergraduate and graduate bibliographies to determine the most popular scholarly journals among students as well as to confirm the "Law of Scattering," a phenomenon that "describes the manner in which articles on a subject are dispersed through the periodical literature" (p.22).Based on the analysis of bibliographies for doctoral students in Spanish and Portuguese over a ten-year period, Gasparatto (2014, p. 88) found that monographs have a "useful shelf-life that is lengthier than scholarly journals".The proposed implications for budget allocation suggests greater historical coverage was needed in monographs, and that 'retrospective' funds needed to be directed towards filling in gaps in the monograph collection.Herther (2009), in addressing the key issues in citation analysis for libraries and librarians, notes that incorporating analysis of student bibliographies as a measure of performance is one metric that can be used to show accountability and to justify funding.
Nearly all studies that we consulted that conducted citation analyses of student work make reference to the perception of students' overwhelming preference for electronic resources over traditional print resources.Davis and Cohen (2001) tracked economics students' citation behaviour across multiple years (1996 and 1999) to show less reliance on books and monographs over time.McClure et al (2011, p. 36), found that 82% of citations were web-based and discussed the role of factors imposed by faculty (including a "minimum required amount of print books") as an influence on the breakdown of resource type selection.In contrast, Gasparatto (2014) discovered that, over the course of a ten-year analysis, use of print monographs was not declining substantially, though the study did reveal a shift to a reliance on electronic access to journal articles.

Citation analysis and instruction
Instructors hoping to assess the effectiveness of IL instruction have relied on student work to show evidence of learning and many have turned to student bibliographies for that evidence.Bennett and Brothen (2010, p. 427) note that citation analysis is often used as a "backward looking measure" for evaluating both instruction and collection development.In their study, the authors utilise citation analysis as a "planning tool for recalibrating library's instruction programme" (p.427).They noted students' difficulty in generating citations (particularly with electronic resources) as well as their confusion over what type of and how many resources should be incorporated in their annotated bibliographies.Ursin, et al. (2004), looked specifically at student usage of resource lists provided to them during instructional sessions, checking students' list of citations against resource guides generated by librarians and based on the assignment addressed in an IL session.In Ursin's study (2004, p. 288), use of resources highlighted by librarians was "consistently very low" (6% of total citations); however, Rafferty (2013) found that first-year medical students were very responsive to resources highlighted during library instruction, citing resources discussed during a session 50% of the time and citing resources linked to in a LibGuide 22% of the time.McClure et al (2011, p. 33) "combined citation analysis with close reading of student research papers" to determine the effectiveness of an online IL tutorial in comparison to in-person instruction.Their findings show that success in finding a variety of sources was not dependent on what form the instruction took.Rosenblatt (2010), in conducting a citation analysis following a library instruction session, found that students were capable of finding a variety of sources and evaluating those sources at an acceptable rate regardless of whether they had encountered a library instruction session; however, when conducting a textual analysis of the final assignment, he found that students were unable to adequately synthesise the sources they had uncovered.
The researchers conducting citation analysis typically rely on rubrics to evaluate student performance.While rubrics varied from project to project, many researchers relied on similar criteria to assess student work, with emphasis placed on variety in document type and currency of sources (Tuñon and Brydges 2005;Edzan 2007;Bennett and Brothen 2010;McClure et al 2011;Flaspohler et al 2007).Tuñon and Brydges (2005) utilised rubrics to assess dissertation resource lists as part of a larger audit of a doctoral programme, while others developed rubrics to allow for both a quantitative and qualitative assessment of bibliographies (Bennett and Brothen 2010;McClure et al 2011;Flaspohler et al 2007).Edzan (2007) generated a scoring rubric that assigned points to students' bibliographies based on the number of citations for various source types.In this coding scheme, the researchers gave additional weight compared to newspapers, magazines, and non-scholarly web sources.In the literature that we explored, we did not find other instances of this type of weighting.

Methodology
In the summer of 2013, our library learned that faculty assessment of core common assignments, to which librarians had previously mapped to IL outcomes, would be dramatically altered and considerably reduced.This change would have a negative impact on the assessment data that the library had previously relied on to chart and document student IL performance.Essentially, although faculty made decisions in the best interest of the core academic division, the resulting change nullified our ability to track students' growth in IL from one year to the next.
While the assessment strategy previously implemented was no longer viable, the need to assess students' IL abilities in order to shape our lessons remained.To that end, the authors reached out to the college's Learning Assessment Director to help gather artefacts for examination.Working with actual student assignments would allow the authors to authentically assess students' success in achieving the college's IL competency.
Champlain College utilises an online ePortfolio assessment platform where students are required to upload common assignments and where faculty grade and maintain a database of assessment Carbery and Leahy.2015 Journal of Information Literacy, 9(1) http://dx.doi.org/10.11645/9.1.1980data.Assessment data from this platform has historically been shared with librarians, although this represents only data for the performance of a class year as a whole.For this project, we were granted access to the platform and the complete set of annotated bibliographies by first year students in the spring of 2013.We were not given access to the individual assessment data generated by faculty for this set of artefacts.This gave us the opportunity to assess information literacy independently, completely uninfluenced by faculty grading of students' overall academic performance.This does mean, however, that for the purposes of this project, we were unable to correlate information literacy performance with overall academic performance.
Reasons for choosing the annotated bibliography assignment for this research were two-fold.Firstly, it is a common assignment that all first year students are required to complete in their second semester.Secondly, as part of our embedded IL programme, librarians tailored a second semester, one-shot instructional session for first years to specifically address the requirements of this assignment.While there have been small tweaks to this session over the last eight years, the lesson itself has largely remained the same over time.The total pool of artefacts was 565, from which the authors pulled a representative sample of 72.We determined that this sample (over 12%) would provide a sufficient basis for completing the project.
To assess student work we utilised a rubric, in line with common practice for librarians and educators in higher education (Hoffman and LaBonte 2012).Librarians at Champlain College, with the support of faculty members, had previously developed the rubric that was chosen for this project, to assess evidence of IL in students' annotated bibliographies.The teaching librarian team originally conceived this rubric primarily for use by the faculty; however, on one previous occasion, both librarians and faculty used the rubric to grade a small sample of student work to gauge differences in each groups' method of grading.For this project, we would be working independently of faculty grading and with a larger sample, though the rubric would remain unchanged.
To ensure that the rubric still represented a viable set of criteria for grading annotated bibliographies eight years after its creation and to calibrate inter-rater reliability, we chose a small initial sample (five in total) to each read, assess, and discuss.After the initial reading, inter-rater reliability was determined to be good, although there were questions raised, which will be discussed below.
Following the initial simultaneous assessment of five artefacts, the remaining 67 were divided equally between the authors and completed separately in three phases.Following each phase, artefacts and assessment data were shared and reviewed to ensure grading was normalised throughout the project.

The rubric
The rubric (see Appendix A) consists of five criteria: presence of a thesis, variety of sources, quality of citations, completeness of annotations, and holistic impression, and can be mapped to Champlain College's IL college competency.In 2003, librarians and faculty developed the IL competency, which represents one of the nine competencies of a Champlain College education.Evaluation of each criterion is done on a four-point scale, with grade 3 and above denoting successfully reaching competency level.When evaluating for presence of a thesis, students were awarded a score of 1 (no thesis present) or 4 (thesis present).
In applying the rubric to the annotated bibliographies that were selected, it was important to ensure that expectations for each of the criteria were clear.This was done in the calibration noted above, during which some questions were raised concerning the finer points of the categories for grading on the rubric.For example, we noted a temptation to weigh the completeness of annotation more heavily in the holistic impression than the other criteria.In addition, the variety of sources criteria, as written, had to be understood in context.We determined that the idea of the 'appropriateness' of the variety to the students' topic allowed room for flexibility when dealing with annotated bibliography topics that draw on a wide variety of eras.

The citation analysis checklist
The citation analysis checklist (see Appendix B) was designed as a metric tool to help us understand how students find information for their academic research.By devising such a checklist, we aimed to track students' item-level citations and compare them against Champlain's physical and online library collections for evidence of library collections usage for academic studies.We also examined each citation to understand whether students discerned from scholarly or popular sources, as well as current and up-to-date publications, or whether they rely on older materials.Finally, the citation analysis checklist also helped the librarians to determine to what extent students use primary source materials for academic research purposes.Each citation from the annotated bibliographies was added to the checklist, and librarians analysed each citation in turn.

Findings
This section outlines the main findings of this assessment project.Using quantitative data from the ePortfolio assessment software, the individual scores of the criteria were assessed.

Thesis
As illustrated in figure 1, almost all students demonstrated presence of a research thesis within their annotated bibliography.A small number (5.5%, n=4) of students did not demonstrate a thesis topic within their annotated bibliography.The median score for the thesis outcome stands at 3.8 on a four-point grade scale.

Variety of sources
Overall, students performed well in finding and using an appropriate variety of sources for their annotated bibliography assignments.Students often made regular use of scholarly journal materials in their bibliographies, and this is illustrated in high frequency of scores of 3 and 4, as illustrated in figure 2. In general, students chose to make use of sources from within a variety of formats, including websites and monographs.The median score for variety of sources stands at 2.4 on a four-point grade scale.

Citations
In total, almost 49% (n=35) of students performed at grade 2.0 against the citations outcome (illustrated in Figure 3).Just over 40% (n=29) of students received a grade 3.0, while 7% (n=5) performed at a 1.0 grade and 4% (n=3) received a grade of 4.0.The median score for citations was 2.4.

Annotations
The median score for the annotation outcome stands at 2.2.The greatest number of 1.0 Grade scores was received within the annotation outcome with 15 students receiving the lowest grade.Almost 21% of students received this lowest score for annotations.A further 43% of students (n=31) received a grade 2.0 in this outcome, while almost 28% (n=21) received a 3.0 grade and almost 7% (n=5) received the maximum 4.0 grade (figure 4].

Holistic impression
As illustrated in figure 5, almost 57% (n=41) of students performed at a grade 2.0 under the holistic impression outcome of the rubric.One-third of students (n=24) received a grade 3.0, while 7% of students (n=5) received the lowest grade and almost 3% (n=2) received the highest grade.The average median score for the holistic impression outcome of the rubric stands at 2.3.

Overall performance
As illustrated in Figure 6, the average median scores by criterion range from 2.2 for Annotation, to 3.8 for Thesis.Elsewhere, students achieved a median score of 2.3 for Holistic Impression while the average median score for variety of scores stands at 2.7.
When we chart (see Figure 7) the frequency of the overall 4.0 grade scores of students as assessed by the rubric, we can see that a 7% of students (n=5) performed lower than an overall score of 2.4 The highest frequency of scores (n=20, 28%) occurs at grade 2.4 while the highest grade achieved (grade 3.8) was received by one student (1.4%).The median overall score received was 2.6 on the four-point grade scale.

Topic selection
The results of our assessment revealed a great deal of interesting information related to students' topic selection.In a large number of cases, students gravitated towards topics that could be considered non-scholarly, from the experience of College Resident Assistants to the community of My Little Pony fanatics known as Bronies.This focus for inquiry can prove challenging for students as there is often very little in the library's collection to support the investigation of these topics.However, in comparison to students with more scholarly topics, these students showed greater familiarity with and deeper reading towards their topics.Often, this meant they made better connections between different sources in addition to analysing them with greater authority.However, regardless of whether their focus was scholarly or non-traditional, students struggled with defining the scope of their research.In situations where students showed greater familiarity with their topic (as typified by many of the non-scholarly topics), they struggled to situate the community in a larger context, neglecting to use scholarly or disciplinary literature to help define their communities.For traditional topics, where students investigated broad historical groups, the annotations rarely gave any indication of an historical focus for the group under consideration; instead, students approached their topics by discussing, for example, all Vikings or all Mayans.

Source selection
The results show that students, on the whole, were successful at identifying relevant sources of decent quality.Because of the diversity and originality of their topics, we also saw students relying on a variety of resource types.Not surprisingly, topics of current interest lent themselves well to electronic formats, and students showed an ability to navigate many 'non-traditional' sources of information, including blogs, listservs, audio (CD or mp3), and documentary video found online.Students exploring traditional topics relied more heavily on scholarly resources (books and journal articles).While these students typically relied on quality scholarly work, the currency of their sources often tended to be an issue.In addition, because their topics often remained very broad, students with traditional topics had trouble integrating broad historical studies with narrowly focused monographs or journal articles.
While students were not expected to use primary source materials specifically, such material types were very appropriate for this assignment regardless of the topics students chose to investigate.While some students did choose to use primary sources, in many cases, the scope of our collection did not support primary source research for students' array of topics.In addition, students may lack knowledge of what a primary source is, where to find primary sources, how they can be used, and why they are important to research.In our study, some students misunderstood the term 'primary source' to mean the main source on a given topic.This finding was a significant one for this study.
Overall, when students engage in academic coursework, it seems that students do indeed make attempt at critically evaluating sources.Many students discussed the concepts of authority, bias, relevancy and currency in their annotated bibliographies.However, this critical approach often lacked sophistication and development.Many students often surmise that good authority equals authors with PhD qualifications, or who had published in any major publishing house.In absence of some of these rudimentary criteria, students neglected to discuss some less obvious considerations, such as why a blogging member of the Brony community might actually be an appropriate source for an annotated bibliography on the popular fan group.It appears that students lacked the ability to think about issues of authority and other evaluative criteria through a contextual lens.Students were unable to add what unique perspective or insight each author brought to their argument, for example.

Overall performance
This research investigates how students find, use and evaluate information and library resources for authentic academic coursework.Assessed against our four-point rubric, students' overall performance equates to a C-grade average.This average grade is where we might expect firstyear students to be in their overall information literacy development.Our embedded instructional programme will allow students to continue on this developmental path, building the skills and dispositions over the remaining three years of their undergraduate studies in Champlain College.

Instructional redesign
These findings have proved invaluable to our team from an instructional design point of view.Challenging our assumption that students needed help finding scholarly materials, librarians were able to shift and refocus the emphasis of our 'COR 120 Annotated Bibliography' IL lesson to be less concerned with finding sources, and more focused on evaluating and synthesising sources.Indeed, reframing our inquiry-based instruction has allowed teaching librarians to engage with students on more nuanced discussion of authority, as well as how their research presents them with an opportunity to take part in a scholarship conversation.While this shift in instruction proves to be subtle, it has allowed us to restructure our lesson towards that which students appear to need greater support with, without completely ignoring the skills and attitudes in which they appear to be performing well in.
More concerning to us, however, was our discovery that students had much confusion around the area of primary source documents.Our assessment effort has clearly highlighted the need for improved instruction in this area.In part due to a changing curriculum, librarians found themselves in the position of having to design a new 200-level (second year) IL session that these same students would later undertake as part of their undergraduate studies.The content and curriculum of this new 200-level course was directly tied to historical subjects, and the course assignment would ask students to research some historical communities.This assessment project has given us a clear indication of the opportunity to address some of the confusion that students feel for primary, historical documents, and in the Spring of 2013, librarians designed an IL lesson for COR 280 that asked students to explore historical primary documents in great detail.This is particularly beneficial for the students assessed in this project, as it is the same cohort of students who had submitted annotated bibliographies that were assessed here.In short, librarians were able to close the assessment loop with the same students.

Implication for collection development
We believe the implications for collection development are manifold.This project gave us insight into what kinds of topics students are investigating and how topic choice impacts their use of our resources.It has also provided a methodology going forward for tracking and analysing students selection habits.It has also confirmed our assumption that students will take an interdisciplinary approach and that we should continue to build a collection that reflects this.However, precisely how we will use this information to implement changes to our collection development policies remains to be seen.Expanding the scope of our online holdings or exploring initiatives such as Patron Driven Acquisitions may have an important impact on how students are selecting, evaluating, and using library resources.

Conclusion
Using rubric-based assessment and citation analysis methods for informing instructional design and pedagogical approach has been very fruitful.By using this two-pronged approach, librarians have begun to understand patterns of behaviour with topic selection, finding information, evaluation, and citation.The assessment effort has also allowed us to discover the areas of difficulty that students encounter with this assignment, which in turn prompts us to consider how we might realign our instructional efforts to address these areas.The citation analysis also helps us begin to comprehend to what extent students use library collections for their academic studies.
This assessment project has had a significant impact on our instructional efforts, with adapted lesson plans and new sessions designed directly as a result.In an effort to check the impact of these changes on students learning, we fully intend to assess future first-year annotated bibliographies in a similar way.This will allow us to determine if our instructional efforts have been successful, and to determine what the impact of these changes have on overall student performance.Furthermore, we fully plan to develop our assessment model beyond the first year annotated bibliography assignment with plans to identify appropriate common assignments in later

Figure 2 :
Figure 2: Frequency of scores received for 'Variety of Sources' criterion, n=72.