They can find it, but they don’t know what to do with it: Describing the use of scholarly literature by undergraduate students

This article discusses a two-part research study. In the first part, the impact of a library instruction (LI) session on the quality of sources cited by upper-division undergraduate students was measured. Student papers and bibliographies were analysed with the use of a checklist in order to determine if students who attended a library instruction session were more likely to cite scholarly sources that addressed their paper topics than students who did not attend a session. Twenty students in total participated in the study and it was discovered that most of the students had no difficulty finding and evaluating sources in order to meet or exceed the professor's bibliographic requirements, including those students who had never attended a library instruction session. However, half of the students in this study had problems synthesising the quality information they found and incorporating it into their papers. This realisation led to the second part of the study. The students’ papers were reexamined in order to describe students’ use of the scholarly materials. A rubric was created in order to determine if the papers exhibited evidence that the students had synthesised the materials they found. Many of the students only summarised the scholarly sources specified by their professor’s bibliographic requirements then went on to make assertions about their topics without making any connection to the scholarly literature they so carefully accumulated. These findings question some of the assumptions about the information literacy needs of undergraduates held by many practitioners, as evidenced by the emphasis most instructional librarians place on teaching students how to locate, retrieve, and evaluate materials from “acceptable” sources. Further research on how undergraduates actually use the information they find can only inform librarians’ pedagogical efforts and may lead to increased collaboration between librarians and discipline faculty.


Introduction
It is commonly held among most librarians, and perhaps content-area faculty staff, that if students can be taught how to evaluate sources they will be able to choose materials that are both suitable for college/university research and address the topic or question they aim to explore.It is also accepted that once students believe in the "inherent goodness" or usefulness of these types of 1 In the United States, most students complete a bachelor's degree at a university in four years.A student is considered an upper-division student during her last two years of study, when the majority of her coursework is within her major.However, the term upper-division doesn't refer to the number of years a student has attended school but to the number of credits they complete.As the majority of students examined take five or more years to complete a bachelor's degree, this term is used in this study to describe students who are in the latter years of their degree.sources, they will choose to use them, especially if they have been taught how to access these high-quality materials using the tools of the modern academic library: metasearch engines; customised widgets; databases; Google Scholar with a connection to a citation resolver, to name a few.It is then assumed that students, possibly with the help of their professors, will be able to synthesise the material contained in these resources through the production of some sort of research-based product, usually a paper.It is also assumed that through this process students gain a deeper understanding of their discipline.

Citation analysis
Much of the bibliographic instruction done in academic libraries focuses on supporting students as they become initiated into the research process.Librarians develop lessons using the best pedagogical practices in order to help students acquire the skills promoted by the ACRL Information Literacy Competency Standards for Higher Education (2000) or the Chartered Institute of Library and Information Professionals Definition of Information Literacy: The Skills (2004).As a rule, librarians focus on teaching users how to locate, retrieve, and evaluate needed information from suitable sources as efficiently as possible, as the acquisition of these skills is seen as a benchmark for meeting these information literacy standards.
A number of academic librarians have used citation analysis of student bibliographies in order to provide evidence that they have been successful instructing students in the information literacy practices.These studies look for a correlation between the information literacy concepts or bibliographic tools explored during a library instruction session and the types of items cited by students as one way to measure the impact these sessions have had on students' behaviour (Barratt et al 2009;Davis 2002Davis , 2003;;Gratch 1985;Hovde 2000;Hurst and Leonard 2007;Mohler 2005;Robinson and Schlegl 2004;Ursin et al 2004;Young and Ackerson 1995).The rationale for the use of this methodology is explained by Hovde (2000, p.5): Examination of student research paper bibliographies provides a flexible, non-invasive, time-efficient assessment forum in which one can quantify actual student behavior (as opposed to self-reported process or attitude) involving library products and resources. . .Whether in conjunction with a demonstration or other lecture-activity combinations, the standard one-shot class session lecture will have its greatest influence on short-term behavior.The 'here is how you do this approach' (and librarians are fully cognizant of its drawbacks) is simply too abbreviated to affect longer-term knowledge integration, attitudes, or motivation.
These citation-analysis studies often describe the types of items cited by students in their bibliographies in order to determine if students have cited the scholarly resources that are considered appropriate for academic work.The idea of scholarly is explicitly defined in Robinson and Schlegl (2004, p.278) as consisting of books, journals, and government documents.Magazines, newspapers, and "low-quality web pages", defined as those without attribution or authoritative authorship, are typically considered non-scholarly.Some of these studies (Barrett et al 2009;Hurst and Leonard 2007;Wang 2006;and Hovde 2000) have found information literacy (IL) instruction to be successful in encouraging students to use scholarly materials, which is seen as evidence that students have been able to evaluate information, and therefore subscribe to the IL definitions promoted by CILIP and ACRL's Standard Three.There is also an interesting body of research claiming that instruction alone does not make a significant difference in the types of items cited by students, but instruction combined with academic penalties for not following a professor's bibliographic requirements significantly increased the number of scholarly sources cited by students (Davis 2002;Davis 2003;Robinson and Schlegl 2004).By contrast, other studies (Ursin et al 2004;Mohler 2005) argue that information literacy instruction had an effect on student behaviour in terms of the types of sources consulted.
In this study, in common with many of those cited here, students' bibliographies were examined in order to describe the types of items they were citing and to ascertain whether these items were from scholarly sources that met their professor's bibliographic requirements and produce university-calibre work.This point of view is summarised by Hurst and Leonard (2007, pp. 1-2) when they state, "Tired of reading and grading mediocre papers, all of which cite flimsy sources, if they cite any at all?It is said that we are living in an 'Information Age.' So, why then, are papers so often lacking in solid, factual information from scholarly sources?… Thus, a path to improved student papers is making students aware of the wealth of resources available to them".However, this study looked beyond the bibliography in order to determine the topics of the students' papers and establish whether the sources cited actually addressed the topics of the papers.The content of the research papers also became relevant when it was decided to examine students' use of the scholarly sources in context.In addition, it is important to remember that in most college/universitylevel classes, the bibliography is not the end product, and that the creation of the research paper, project or presentation, i.e. the end product, is usually seen as a means of demonstrating the acquisition of competencies set by the requirements of the relevant discipline.

Rubrics
Other researchers trying to assess the impact of library instruction on student work have noted the limitations of citation analysis when trying to fully understand students' acquisition of information literacy skills (Diller and Phelps 2008).These researchers have turned to rubrics and ePortfolios to gain a greater understanding of their students' abilities.A rubric is a grading tool that describes how an assignment will be assessed.It is a form of criterion-referenced assessment, consisting of criteria that "divide the assignment into its component parts and provide[s] a detailed description of what constitutes acceptable or unacceptable levels of performance for each of those parts," (Diller and Phelps 2008, p. 3).A rubric can be used to communicate the instructor's expectations to students and can also be used to simplify the grading process and to make grading more consistent.Materials evaluated with rubrics are assessed in terms of whether the work meets the criteria delineated by the evaluator.In criterion-referenced assessment, every student in the class can receive full marks if he/she meets all of the professor's expectations.This method of assessment can be contrasted with normative assessment, which would require the grader to compare students' work on the same assignment.In a norm-referenced system grades would be assigned based on how well students performed in comparison with each other.
Rubrics and other criteria or standards-based forms of assessment are common in primary and secondary education.Both Oakleaf (2009) and Helvoort (2010) have documented the use of rubrics and portfolios in information literacy assessment and recommend best practices for their use.In addition to providing more transparency to the evaluation process, rubrics used to assess student work generate descriptive data that is beneficial to the evaluators, both librarians and faculty staff.A rubric can provide structure to a qualitative evaluation of student work, as well as a means for conducting a quantitative analysis of student scores when competency levels are assigned point values.The creation of a rubric can also inform an educator's practice.As noted in Oakleaf (2009), while creating the rubric, librarians and their collaborators are forced to reflect on and articulate their instructional goals.After the rubric has been created, these goals can be enumerated in a language that can be understood by a number of stakeholders.

Methodology
This paper summarises a two-part research project that utilised both citation analysis and the use of a rubric to evaluate the students' papers.The initial study attempted to measure the impact of a library instruction session on the types of items the students cited in their research papers in order to determine whether those who attended a library instruction session were more likely to cite scholarly and content-relevant materials than those who did not attend a session.The methodology employed to ascertain the topic of each paper led the researcher to notice that many of the students did not seem to be synthesising the materials they found by incorporating ideas from the literature into their papers.This realisation led to a second analysis of the data in order to describe how students were using the scholarly literature they found.A sociology professor teaching two sections of the same upper-division course, Sociology of Families, encouraged her students to participate in the study.In the syllabus, students in both groups received the same information about the paper which specified the professor's bibliographic requirements:

Re-Examination Paper --20%
In assignment number two you examined your family of origin2 .In this assignment, you are to re-examine your family of origin given what you have learned in this class.You must cite three articles, one outside book, and one presentation text in this analysis, incorporating theories and terminologies in your analysis.At the end of your essay, make sure to note how your perception of your family has changed, if at all.
Students in the experimental group attended a library instruction session within their class.The session, delivered by the author of this paper who conducted the research, was taught in a computer lab where students were able to practice using the library's resources in order to complete research for the paper.The librarian showed the students how to access the library's catalogue and how to conduct simple keyword searches in Gale Virtual Reference Library, Social Sciences Full Text, and Sociological Abstracts.The librarian also briefly discussed how to determine if materials were scholarly and how to use an article's abstract to decide if the article related to a research topic.Students in the control group were given the assignment in the syllabus and expected to find the appropriate materials on their own.At the end of the Spring 2008 semester, students from both groups were asked to volunteer their papers for the study.Submission was done on a voluntary basis simply because the researcher was not the students' instructor and would not normally evaluate their work.Whilst the dependence on volunteers may have influenced the outcome of the research, it was deemed necessary by the researcher in order to comply with the institution's ethical rules when dealing with human subjects in research.Twenty students voluntarily submitted their Family Re-Examination Papers and bibliographies.Eleven of the students belonged to the experimental group and nine students were from the control group.A research assistant removed all identifying information from student papers and assigned a number to each paper.

Developing a checklist
For the initial study, the researcher used a checklist to evaluate the citations in the students' bibliographies using criteria from the ACRL Competency Standards for Higher Education (2000) and the professor's bibliographic requirements.
A codebook was created for the checklist used in this research in order to ensure that the data was evaluated consistently.For the purposes of the study, the codebook defined scholarly articles as those published in journals determined to be peer-reviewed by Ulrich's Periodicals Directory.A book was considered scholarly or academic if it had a WorldCat record that indicated that the book included a bibliography or reference list.The criteria for determining if websites, podcasts, vodcasts, and other A/V materials were scholarly or appropriate for the assignment were based on those used by Robinson and Schlegl (2004).These materials were determined scholarly for the purposes of this study if: • the item was a reprint of an article or chapter that appeared in a scholarly or peer-reviewed publication as determined by the criteria stated above OR • the domain name was .govor .eduAND • an author could be determined AND • a bibliography or references were provided to support assertions made in the item.
The topics covered by the papers were determined by reading the first page of each paper.If the topic was not revealed in the first page, the researcher would continue reading until a topic could be determined.If the paper's topic could not be ascertained, the paper was excluded from the study.Six topics were presented as part of the checklist.These topics were generated from the course's syllabus and the WorldCat records for the six books assigned for the class.The topics listed were: migration; same-sex parenting; single parenting; racial or class discrimination; balancing home and work; and marriage, although the researcher also had the option of selecting multiple topics from the list or including additional topics to the list.A common topic emerged from this analysis as several students decided to write about when they or a sibling "came out" or revealed their sexual orientation to their families.
The checklist enabled a comparison between the topic of the sources cited and the topic of the paper in order to determine if students chose items that were relevant.The topic of a cited source was determined by reading its title.If the title was not sufficient to determine that the item's topic related to the paper's topic, the researcher examined the context in which the item had been cited and looked for following information about the source: • Books: Look at the bibliographic record for the item in WorldCat • Articles: Look at an abstract for the item • Other items: Go to the actual item as identified in the bibliography, such as web pages, podcasts, or blog entries If the topic of the source was still unclear, or the item did not appear to have been cited to support the student's argument in relation to the topic covered by the paper, the item was deemed irrelevant and this information was recorded on the checklist The researcher coded all of the student papers and bibliographies, although the reliability of the checklist was tested for inter-rater reliability with the help of a second reader.A random subsample (three papers) of the data was selected.The researcher and the second reader separately evaluated these papers and bibliographies using the codebook and checklist.The results were compared, and inter-rater reliability was calculated based on percentage of agreement (Boyzatzis 1998, p.154): Percentage of Agreement = no. of times both coders agreed no. of times coding was possible 200/(109 +109) = 0.917 Based on 92% agreement between the researcher and the second reader the checklist was considered reliable.

Developing a rubric
For the second part of the study, a rubric was used to evaluate the same set of papers examined in the first part of the study in order to categorise the papers based on evidence of the students' ability to synthesise the sources they retrieved and use them to support their arguments.The rubric (shown in Figure 1, below) was created using processes described by Stevens and Levi (2005) and Boyatzis (1998).As the role of the rubric was to help the researcher identify evidence of synthesis, the term synthesis was defined in terms of the professor's assignment and the ACRL Information Literacy Competency Standards Three and Four (2000) reflecting the ability of students to read and understand new material and incorporate it into their knowledge base.Synthesis would be evident in papers where students were able to make connections between the literature and their own ideas about their topics, their personal narratives, and the examples they chose to reveal from the interviews they conducted with family members.
The researcher then listed the criteria needed to meet the professor's main goal for the course, namely that students begin thinking like sociologists, as well as the competencies described in the Association of American College and Universities VALUE rubrics (2007) that they needed to possess as upper-division students.The papers were reread and separated into broad groups: those papers that showed examples of the ability to synthesise the literature and those that did not, whereas the papers that showed inconsistent evidence of synthesis were placed in a separate category.This enabled the researcher to determine that a three-part rubric would need to be created.This iterative process for developing a code, in this case a rubric, is explained in Boyatzis (1998).
The rubric was constructed using concepts and ideas from the professor's syllabus, the ACRL standards, and the VALUE rubrics.The rubric employed the three categories used by Diller and Phelps (2008) to describe the student's ability to synthesise scholarly materials: Emerging, Developing, and Integrating.It was postulated that papers in the emerging category would not show any evidence that the student made connections between the literature and the paper's topic.Papers in the developing category would show evidence of few connections, but the process of connecting the scholarly literature to the topic would not be consistent or sustained.Papers falling into the integrating category would show evidence that the student was meeting the professor's goal for the course by consistently connecting the scholarly literature to the paper's topic throughout the paper.As in the early stage of the study when the checklist was being devised, the researcher coded all of the papers and the rubric was tested for inter-rater reliability with the help of a second reader.A subsample of three papers was selected and the researcher and the second reader individually evaluated the papers using the rubric.The findings were compared and reliability was measured using the same method of determining percentage agreement that was used to determine the reliability of the checklist (Boyatzis 1998, p.154).There was an 83% agreement between the researcher and the second reader and therefore the rubric was considered reliable.The papers were reshuffled then reread by the researcher.As shown in Figure 1, the criteria listed in the rubric were used to categorise the papers based on the students' ability to synthesise the literature.Following from this, the researcher went through the papers one last time, outlining them and noting instances where students made connections between the scholarly texts and their personal life experiences or those of their interviewees.In addition, the students' expanded examination of the sources through comparison or contrast and their critical appraisal of the findings from these sources was taken into account.

Results
Eighty-five percent of the students were able to meet or exceed the professor's bibliographic requirements, including five students who had never attended a library instruction session.Surprisingly, the three students who could not meet the professor's requirements were part of the experimental group and therefore had attended a library instruction session.

Figure 2: Number of students citing items on topic
Overall, the majority of the professor's students were able to find scholarly materials on their topics whether they attended a library instruction session or not.As mentioned earlier, the only students who had problems meeting the professor's bibliographic requirements for the use of scholarly materials or cited 50% or fewer resources related to their paper topics were those students who attended the library instruction session customised for this assignment, and two out of these thee students also attended library instruction sessions prior to this class.These findings suggest that in the case of these students library instruction had no impact on the students' ability to discover or evaluate resources.Unfortunately, the examination of this negative impact goes beyond the scope of the research presented here, although the author is planning to explore findings such as this in a future study.Ninety percent of the students produced bibliographies that contained mostly scholarly materials (i.e.70% or above were scholarly sources) and most of the students used a greater number of scholarly sources than the five required by the professor.In addition the number sources that were deemed relevant to the topics were also high (80% or above).By contrast, only 10 of the twenty students who participated in the research showed evidence of the ability to integrate the scholarly information they discovered into the arguments they made in their papers.A detailed examination of how students used the information they cited now follows. .

Emerging (20%)
The papers in this category showed no evidence that the students were able to analyse, synthesise, or use the literature they selected.These papers began with an introductory paragraph or sentence and the scholarly literature, consisting of articles and books, was summarised, with a paragraph dedicated to each item cited.Interviews with family members and narratives about the student's life were appended.There were never any connections made between the texts and the student's experiences.In one paper, a student summarised three research studies in three separate paragraphs.The student described each study's research question, the methodology used, and summarised the findings.This student never compared or contrasted the findings of one study with those of another study, nor did he/she discuss the implications these findings could have in terms of his/her own topic.This treatment of the scholarly literature as a prelude to the discussion of the student's ideas or family history was common across all the papers in this category.Another paper, entitled The Economy, My Family and Me, also began with a summary of four of the sources cited.Each source was given its own paragraph.The topics addressed in the literature were the tension between work and family, the impact of financial difficulties on adolescents, and an evaluation of Reaganomics.All of these topics seem to relate to the student's overall theme, although this was not made explicit through the student's writing.This student proceeded to describe his/her family's views on capitalism, unions, and the assessment of the impact various U.S. presidential administrations had on the economic status of the American family, supporting his/her assertions with quotes from family interviews.This student concluded: That is the capitalistic society we live in today.There are laws against perjury and fraud, but is withholding the truth the same as deception?There are laws against theft and burglary, but are there not laws against withholding earnings?What are we working so hard for?Do we not deserve a living?Who represents us?What is stopping us from taking what we need?As my mother said, the American economy is killing the family, but it is not dead yet.
While the theme addressed in this conclusion had some connection to those expressed in the literature cited, it was crafted to represent a summation of the thoughts expressed in the narrative.This treatment of the scholarly literature as completely separate from the argument crafted by the narrative is indicative of the papers in this category.

Developing (30%)
The papers in this category began to show evidence of the students' ability to connect themes and ideas from the literature with those expressed in their own narrative.But these connections were not consistent or sustained.In addition, the topic of the paper was usually introduced by more than one paragraph and a single connection to the literature usually occurred in the introduction.Subsequent scholarly sources were often summarised, while the use of peer-reviewed journal articles was sometimes described.For example, one student wrote, "While going through peerreviewed journals, I found numerous articles," and later continued, "After sifting through several peer reviewed journals I found a diamond in the rough." Overall, a maximum of three connections were made between the scholarly literature and the life experiences of the students or interviewees.One student made the connection between the border crossing depicted in Victor Villasenor's Rain of Gold (1992)3 and her mother's personal experiences, "The tragic scene presented in the book Rain of Gold reminds me of one of the experiences my mom mentioned she experienced while she and my dad were crossing the border."As discussed earlier, the students were required to cite at least five sources in order to meet the professor's bibliographic requirements and all of the students in this category cited five or six sources and therefore fulfilled this requirement.Occasionally, points made in one or two articles were used to support each other, but this did not occur in all of the papers, therefore these students made connections between the literature and their narratives but not consistently.
All of the papers had conclusions but these made no connections to the literature, as shown by this example: Parents still try to arrange the marriages today but couples are given time to see if they would like to continue [sic]

Integrating (50%)
Papers in this category showed evidence that students were able to synthesise the materials they found in order to "think as sociologists."In the first or second paragraph of the papers, connections were made between the literature, the topics covered by the papers and the students' life experiences and these connections were continued throughout the papers, providing evidence that students were able to apply the ideas and themes expressed in the scholarly literature in order to describe the events depicted in the narratives they told about their own families.As one student wrote: In one journal article, they argued that mothers are often disadvantaged when it comes to negotiating with their spouse [sic] over who should get a break from work and family demands because of the greater sense of responsibility for taking care of children (Nomaguchi, Milkie 2005:7).I took this point in particular to heart for not only did I experience lack of time to spend with my mother, I also witnessed my mom experience a lack of time for herself because I could tell that she constantly had demands beyond her ability to accomplish comfortably.
All papers in this category made at least one comparison between the scholarly resources and, more importantly, five of the students contrasted the research with examples from their own life experiences or those of their interviewees.This point is clearly shown in the following example, although in this case the full reference is not given: "Research indicates [that] less educated parents might shift their attention away from school because they feel inadequate when helping their children with homework (DePlanty et. al.).Although at times my mom and dad felt helpless they tried hard to help us with our homework.They never gave up because they knew education was important." In the papers' conclusions, broad themes from the literature were usually summarised and these themes were compared to the students' life experiences and those of their interviewees.As one student concluded: Coming out to one's parents can be one of the most life changing and anxiety provoking experiences that a youth will go through.

Conclusions and Further Questions
This study's findings suggest that upper-division undergraduate students can conduct effective keyword searches to discover literature on their topics irrespective of whether they have attended library instruction sessions.These students are also able to evaluate the materials they have found in order to determine that they have chosen items that meet their professor's bibliographic requirements.However, many of these students appear to have difficulty using the information they have found in a critical way.Half of the students involved in this study (20 in total) provided little or no evidence that they derived any benefits from the literature they were required to consult.
Shouldn't we, as instructional librarians, be concerned about students' abilities to use the information they have discovered?This author argues that, by analysing the work products of our students, we can reveal their misconceptions about the research process and why the "literature" is cited in academia.By collaborating with faculty staff, we as librarians can also identify differences between faculty conceptions of the purpose of research and those of their students.If it cannot be assumed that students understand why certain types of sources are used or even why a learned person reads existing research, then how do we know that students are learning by conducting research and writing papers?
The author is planning future investigations into students' use of scholarly sources by adding student interviews to the methodology so that questions raised by the current research, but that go beyond the scope of this study, can be explored.These questions can be summarised as follows.
Why are students not synthesising the literature?Is it because they find it difficult to understand the materials or do these students actually incorporate the information into their knowledge base but do not know how to connect the concepts in the literature to their own ideas due to poor writing?Is there a correlation between the ability to synthesise the literature and the grade awarded to the project/paper?Most importantly, the issue of the impact of library instruction sessions will be examined in a future study particularly as the current research has shown that some students perform badly despite having received some information literacy training.This study's findings led this researcher to reconsider the information literacy needs of students at various levels at her institution and to change her practice by introducing a new way of scaffolding the instruction of information literacy concepts in her library instruction sessions.Time previously spent on ensuring that students practice keyword searching will in future be allocated to modeling the synthesis of disparate sources and to dissecting the work of experts to see the purpose the literature serves in scholarly work.

Figure 3 :
Figure 3: Number of students meeting or exceeding the bibliographic requirements.

Figure 4 :
Figure 4: Students' use of the scholarly literature relationship and see if marriage is to be considered.For me there are more choices today compared to my mom's generation.Arrange[sic]marriages can still be successful today if given enough time to know each other.My cousin Sonal is an example that arranged marriage can work however my experience was different because I got a chance to know and love my husband before I got married to him.