Using a flipped classroom to embed information literacy skills training into academic studies

"By ‘open access’ to this literature, we mean its free availability on the public internet, permitting any users to read, download, copy, distribute, print, search, or link to the full texts of these articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose, without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. The only constraint on reproduction and distribution, and the only role for copyright in this domain, should be to give authors control over the integrity of their work and the right to be properly acknowledged and cited.”


Introduction
In January 2017, King's College London ran a module entitled 'Contemporary Issues in Neuroscience' for the first time as part of its BSc Psychology degree.The module, open to second year students, offered a departure in approach from the previous modules.This was necessary because a) the nature of the content meant that the module did not require students to learn vast quantities of neurobiology but rather critique emerging applications of the discipline by engaging with traditional academic sources and more general sources, b) the students begin their research project shortly after this module and it was felt necessary to further develop their skills as an independent learner, in particular, in the area of information literacy and c) with student numbers increasing, I wanted to develop a method of delivering this teaching which could scale-up in future years.Based on these requirements the module was designed to include three flipped classroom sessions to teach database searching, citation search and how to keep up to date.The aim of this study was to understand the student experience and perception of these flipped sessions.The findings provide a foundation for improvements to this module and feed into the development of other modules.

Review of the literature 2.1 What is the flipped classroom?
The flipped classroom is a pedagogic approach where students typically view online resources, such as videoed lectures or presentations, in advance of contact time, which is instead used to complete learning activities based on the material taught in the online session.The approach was originally made popular in the US secondary education system (Bergmann & Sams, 2009) but its popularity has spread into higher education (HE).Here, it is an example of the increasingly popular blended learning approach where students receive teaching through a combination of face-to-face sessions and by completing activities outside of class, using a range of technologies (O'Flaherty & Phillips, 2015).It is suggested that the flipped classroom fosters student ownership of learning through the completion of preparatory work and the increased interactivity during the actual class time (O'Flaherty & Phillips, 2015).It is also thought to have the advantage that it allows students to work at their own pace, and flexibly.From the perspective of the university, flipped classrooms have the potential to offer a costeffective way of providing a student-centred curriculum in the face of increasing student numbers.
A recent systematic review of flipped classrooms in HE found that most of the published case studies originate from the US, with none from continental Europe and only one from the UK (O'Flaherty & Phillips, 2015), suggesting that the practice is still not widespread.The studies identified used a range of preparation resources, including pre-recorded lectures, screencasts, annotated notes and captured videos including interactive videos from online repositories such as the Khan Academy.The classroom sessions were equally diverse and included case-based presentations, team-based discussions, panel discussions, expert-led discussions, role-plays and student presentations, discussions and debates.It is generally recommended that within the context of HE, class time focuses on knowledge application (Pluta, Richards & Mutnick, 2013) and that appropriate use of flipped classrooms can result in improved student satisfaction (O'Flaherty & Phillips, 2015).

Developing information literacy skills
One area where students benefit from specific training, irrespective of the discipline they are studying is information literacy (IL), i.e. the ability to recognise what information is needed and be able to locate, evaluate and use it effectively (Association of College & Research Libraries, 2000).Kasowitz-Scheer & Pasqualoni suggest that students face a significant challenge from the sheer volume of information that is currently available, and which is being added to daily.They may struggle to use information efficiently and effectively (Kasowitz-Scheer & Pasqualoni, 2002), meaning training in this area is becoming even more important.
In line with many universities, students at King's College London receive library training by librarians during their induction week in their first year.Induction activities of this kind have been discussed in the literature for decades (Barton, 2017) and despite significant developments, there are still issues of low student engagement (Barton, 2017;Carpmael, Morgan & Nichols, 1992).This is not necessarily surprising given the major transition students undergo as they settle into university (Maunder, Cunliffe, Galvin, Mjali & Rogers, 2013).The demands of this transition, coupled with the fact that student generally show low engagement in study skills development (Burnett & Collins, 2007;Thompson, Kardos & Knapp, 2008;Verlander & Scutt, 2009;Wingate, 2006) mean the value of these sessions is often limited, with students returning to the library or academic staff for help with basic ILskills later in their studies (Barton, 2017), normally when assessed work requires the skills to be demonstrated.Additionally, when delivered separately by librarians during induction, the material is not fully integrated into the students' academic curriculum, meaning they can often struggle to see the relevance of the skills.An integration is deemed preferable because it links IL into the rest of the students' experiences (Orr, Appleton & Wallin, 2001;Snavely & Cooper, 1997).Even if integrated into the curriculum, other challenges exist to making the skills training effective.These challenges include ensuring students are suitably motivated to learn the skills and designing appropriate assessment to evaluate whether students have mastered the necessary skills (Grassian & Kaplowitz, 2001).Given that flipped classrooms are thought to improve student engagement and satisfaction, they may offer a suitable approach to delivering IL skills training.

Using the flipped classroom to teach information literacy
The idea of using a flipped classroom approach to information literacy instruction is not new.Indeed, a review of library training for 140 libraries found that 107 (76%) provide instructional videos on their website.However, of these, only 2 (2%) clearly instructed users to watch instructional videos before attending a library research workshop, suggesting the flipped structure is often optional or implicit (Obradovich, Canuel & Duffy, 2015).Give that students are often unfamiliar with this classroom approach and do not realise they will need to use the skills from preparation material in the face-to-face class, it is suggested that clear and consistent expectations are required for flipped sessions if they are to work (Stonebraker, 2015).
Several studies have evaluated the delivery of sessions where expectations were made clear, some as standalone library training and others integrated into the overarching curriculum.For example, Rivera compared a traditional 7-week standalone course by librarians with a flipped version and found greater improvements in assessment scores on the course in students who had undertaken the flipped version (Rivera, 2015).Cohen used flipped sessions to deliver a library-discipline faculty collaboration on information literacy instruction for a business management program.The results showed that 70% of the students rated the flipped classroom approach as satisfactory.They also found that the quality of students' work improved, and the students' receptive response to the flipped structure increased the teacher's confidence (Cohen, 2016).However, increases in test scores with flipped sessions may not be equitable for all students, with some research finding that whilst all groups of pupils improved on post-tests following flipped teaching by librarians, minority groups increased more (Stonebraker, 2015).
Whilst this study did not further investigate why the minority groups experienced greater benefit, it seems probable that the ability to work flexibly and at their own pace could be important.

Despite these positive results
, not all studies have demonstrated flipped classroom approaches as successful in terms of assessment outcome.Goates et al. compared search statement development between traditional lecture and flipped instruction sessions lead by librarians in standalone training (Goates, Nelson & Frost, 2017).They found that students in lecture sessions scored significantly higher on developing search statements than those in flipped sessions.Other studies have found no differences between traditional and flipped approaches in terms of test scores or other assessment.For example, Brooks found no differences between assessment scores following traditional and flipped training on a librarian-led course (Brooks, 2014).Similarly, Carroll et al. (2016) examined whether a flipped classroom would improve undergraduate health sciences students' abilities to find, evaluate and use appropriate evidence for research assignments.They found that whilst students learned information literacy concepts, they did not consistently apply them (Carroll, Tchangalova & Harrington, 2016).Although these studies did not indicate significant benefits to the flipped approach in terms of immediate assessment outcomes, student perceptions were positive.In the study by Brooks (2014) students were satisfied with the teaching on video and whilst they did not show a preference for it over face-to-face teaching, 79% would recommend future students watch the online videos and attend the face-to-face session compared to 10% recommending either of the approaches alone.Similarly, the work by Goates et al. (2017) found that student evaluations indicated a strong preference for pedagogies that incorporate elements from both lecture and flipped methodologies.Furthermore, it is not just students who view the experience positively, with library and academic staff also having overwhelmingly positive experiences (Sondergeld, Derrington, Howard, Harden & Reid, 2017).
The mixed evidence reported here is not uncommon with pedagogic approaches using technology, where student preferences and assessment outcomes can move in opposing directions.However, much of the research to date has focused on immediate quantitative outcomes, or occasionally, general module assessment, rather than the qualitative experience of the student.In the present study, I investigate the impact of a flipped classroom for teaching information literacy skills embedded within a module by considering the reflections of the students as well as their performance on specific IL assessment and broader module assessment in which they must apply the IL skills to achieve the learning outcomes.All assessment, including the reflections, were summative assessment, a factor likely to encourage student engagement.

Flipped sessions
Teaching took place over ten weeks, in line with normal term times.For the first five weeks students had weekly lectures and seminars, an approach to teaching that they had experienced in every previous module studied as part of their degree.Each week had a distinct topic focus: neuroimaging, neuromarketing, neurolaw, neuroscience and addiction, and neuroeducation.After these five weeks, students had the opportunity to select from one of four neuroscience essay topics and submit a formative essay plan during Week 6.This plan was restricted to 500 words in length and could include an outline of the main sections of the essay.Feedback was provided within three days to ensure students had feedback in advance of the first skills session.The remaining weeks consisted of flipped workshops and independent study.For each flipped workshop, taking place in Weeks 7, 8 and 9, ahead of assessment submission in Week 11, students were instructed to watch a narrated presentation in advance of a one-hour face-toface workshop in the library led by the module leader and facilitated by a graduate teaching assistant.If students came to the face-to-face session without having completed the online work in advance they were asked to complete it in the session and then join the next face-to-face slot.After the first session, all students reported having completed the online preparation in advance of the face-to-face session.There were also some additional preparation tasks.A brief description of the preparation work is provided in Table 1.different results for the same search In addition, they were required to locate at least one paper to conduct a citation search on in advance of the face-to-face session.

Keeping up to date
After viewing the 15-minute presentation students should be able to:  Understand the importance of keeping up to date with information sources  Use appropriate methods to keep up to date with publications in ejournals such as table of contents alerts; databases searches using search alerts, citations using citation alerts and websites by using tracking alerts and RSS feeds  Decide which methods are appropriate for specific topics In addition, students were required to come to the final face-to-face session with details of an e-journal for which they could set up a Zetoc Alert; a database search for which they could set up an alert; a paper to use for a citation alert and a website to track or an RSS feeder to set up.There were three pieces of assessment within the module, two of which directly related to the flipped sessions, focusing on skills development.Firstly, students had to keep a record of their approach to applying the skills learnt to their chosen coursework question.This record, (shown in Table 2), contributed 25% to their overall module grade.A further 5% of their overall grade was determined by a brief reflection, in which they were asked to draw on Gibbs Reflective Cycle (Gibbs, 1988) and describe their skills in finding information before and after the teaching and assessment on the module.For both pieces, the students were encouraged to begin completing them during and after the skills sessions and then finalise them after the third session.The final piece of assessment, designed to assess their academic achievement in neuroscience applications was a 2500-word neuroscience essay which determined the remaining 70% of the module grade.
Table 2: A copy of the Skill Development Record students submitted as part of their module coursework

Qualitative analysis of skills reflection
Twenty-seven (56%) of the students studying the module provided consent to use their assessment for this study.The reflective piece was reviewed blindly by the author, meaning they were unaware of the student's identity and their performance on any component of the module.Thematic analysis was conducted using a six-stage process (Braun & Clarke, 2006, 2013).Quotes are provided as validity of evidence (Mays & Pope, 1995).Punctuation was added to unambiguous quotes, spelling mistakes corrected, and where necessary, words added in square brackets for clarification.Multiple quotes from one person were treated as a single comment to avoid over-representation of an individual.Finally, an independent researcher with expertise in qualitative research reviewed the reflections and analysis.

Quantitative analysis of module results
For students who provided consent to use their data, the grades on the different components were analysed.Specifically, all grades were converted to percentages and correlational analyses were conducted to examine whether the grades on the skills development record State the two databases that you chose to search from those available at KCL library (2 marks) and give your rationale for choosing each of them in terms of how your chosen databases specifically help you to answer the topic question set (2 marks).
List the search terms that you used, writing alternative terms in brackets after the relevant term (4 marks).Underneath your list of terms, explain how you combined them to conduct your search.You should list the different combinations in order of use to illustrate a logical progression to your search from a basic search to an improved search (4 marks).NOTE: You should do this for each of the two databases that you have used.
State one appropriate limitation that you applied to limit your search results in each database (2 marks).Describe the effect it had on your outputs for the two databases (2 marks).
Provide a full reference in APA style for the paper you conducted your citation search on, which you should have found by completing a database search (1 mark).
State which citation index you used and how many citations were found for the above paper (2 marks).
Provide a full reference in APA style for the most recent paper to have cited your paper of interest (1 mark).
Comment on whether the most recent paper to have cited your paper of interest is relevant to the original search you conducted to find your paper (2 marks).
State one way in which you could keep up to date with this particular field based on the material provided in your final skills session (3 marks).correlated with the essay grade and how the two correlated with the final module grade.It was hypothesised that there would be a significant correlation between the two components and that both would correlate heavily with the final grade.

Results
Four themes were identified during the qualitative analysis: 1) Challenging previous understanding of information literacy skills 2) Identifying specific new skills 3) Recognising the transferable nature of the skills for future practice 4) Changing confidence.

Theme 1: Challenging previous understanding of information literacy skills
There were mixed reports of how students perceived their understanding of these skills prior to the sessions.The majority felt that they had some understanding of database searching, from their previous studies.However, most students were relying on Google Scholar as their only source of information and the sessions challenged this view: "Developing database searching strategies has allowed me to understand that using Library Search Services and Google Scholar is insufficient to detect the most relevant articles." "Prior to this module I would only utilize Google Scholar to find supporting research…I have realised how limited my options were." Even where students were confident in their previous abilities, there was evidence of a change in mind set: "Before I attended the sessions, I felt that my current method of conducting searches was perfectly valid, and that it was unnecessary to learn a more complex method […].During the sessions, I began to understand why it is necessary to use the advanced search." "By ignoring acquiring these skills I didn't realise how I was hindering my work."

Theme 2: Identifying specific new skills
Despite students having some previous experience of database searching, many reflected on learning more about this skill, in particularly, applying limitations to searches and the value of this: "The possibility of filtering articles […] was very useful.I had never considered these limitations on my searches [before]." "I found implementing the database limits most useful [...] before this, I would have read through the abstract of every search result to identify relevant papers, so learning this skill saved me time." However, students also reported the acquiring skills in, and valuing, citation searching:

"Citation searching interested me the most … [as it] allowed me to assess the importance of a paper."
A few students referred to the third session in terms of new skills developed: "Before the sessions I was unaware of the variety of methods can that be used to keep up with research." 4.3 Theme 3: Recognising the transferable nature of the skills for future practice A few students made explicit reference to the transferable nature of the skills taught and how they had applied them to concurrent modules: "I feel that my skills have improved over the course of this module, for collecting information for my essay, as well as for coursework for other modules." However, more commented on how the skills developed would affect their future studies.A reduction in the time taken to find suitable sources was flagged as benefit by several students: "Rather than sifting through pages of irrelevant articles and missing out on many relevant ones on database such as Google Scholar, I am now able to access those that interest me a lot quicker." "The information literacy skills part of this module has undoubtedly made me more efficient when searching for online resources." By contrast, some commented that they would spend more time designing effective searches in future, to make their use of time more efficient overall: "In future, I will ensure to spend a lot of time prior to searching to identify relevant search terms, synonyms and limitations […] similarly I will equally make sure to set up citation/database alerts for multiple journals and papers." "Using these new methods can make the article searching process more time efficient…These skills will definitely help alleviate some of the stress that such a large assignment would bring about." Students also believed that by applying the skills they had learnt, they could increase the quality of their work: "For my future assignments I am going to use the skills I have developed in these sessions to maximise the quality of my future work." "These skills have allowed me to produce high quality searches for this essay and for the coursework required in other modules, greatly benefiting my psychology skillset now and in the future." Students also suggested that following the training they would diversify the tools they used, with a move away from relying exclusively on Google Scholar: "Prior to these sessions I would automatically turn to Google Scholar for finding my papers however, I now feel I could and will use scientific databases." Some also suggested that the keeping up to date skills were more useful for longer projects with several citing their dissertation as a place to apply this skill: "I believe alerts would be particularly useful for long term literature searches." Finally, other students offered general comments indicating they valued the skills learnt and felt they would impact on future practice: "These skills are essential to possess, and have changed how I approach my research."

Theme 4: Changing confidence
Some students referred to their levels of confidence changing following the training sessions.This applied to both database searching and citation searching, which a number had identified as a completely new skill: "I now feel more confident searching for relevant articles and journals."

Relating skills development to academic achievement
The grades for the different components are shown in Figure 1 and indicate that students tended to score highest on the skills development record with the average score being 80%.Overall, the average module grade for the consenting students was 72%.This is identical to the grade of the whole cohort, indicating the sample consenting to participate in this research were representative of the whole student population on the module in terms of performance.It was hypothesised that the performance on the skills development record would correlate with the essay score, but this was not the case (r=0.236,p=0.226).There were strong correlations between the skills development record (r=0.646, p<0.001) and the essay (r=0.876,p<0.001) and the final module result.This is expected given how the module result is calculated.

Discussion
This study described a flipped classroom approach to information literacy instruction.Unlike previous studies, the focus was on understanding what the students felt they had gained by analysing their reflections rather than conducting a pre-and post-test analysis.However, some quantitative analysis was conducted, which is discussed first.The analysis revealed that there was no correlation between student scores on the skills development record, which most scored highly on, and their essay score.Given the face-to-face elements of the flipped sessions were entirely focused on applying the skills to the coursework essay topic, this is surprising.However, it is not unprecedented; Carroll et al. (2016) examined whether a flipped classroom would improve undergraduate health sciences students' abilities to find, evaluate, and use appropriate evidence for research assignments.They found that whilst students learned information literacy concepts, they did not consistently apply them.Our skills development record certainly indicates students learnt the skills but perhaps they failed to apply them fully in the essay.This is partially supported by the qualitative analysis where students referred to the keeping up to date material being more helpful for longer-term research.Indeed, teaching this material in Week 8 for a submission in Week 11 means there was minimal time for database or citation alerts, for example, to produce significant results for the student.The lack of impact of the final sessio, may have reduced any potential correlation between the grade on the skills development record and essay.This could be further explored in future studies through semi-structured interviews with students about applying their skills.
The qualitative analysis indicated that students valued the skills they had learnt.Although this is in line with previous work (Brooks, 2014;Carroll et al., 2016;Cohen, 2016) it should be noted that the positive experience came despite the sessions challenging their previous approaches to database searching, with some students feeling that they had already mastered this skill only to realise how rudimentary their practice was.This tendency for students to overestimate their skills in an area, where in fact they lack abilities, is referred to as the Dunning-Kruger effect (Kruger & Dunning, 1999) and has been found previously for IL skills (McKeever, 2013).The result is that students are often disengaged with the teaching because they feel they already have the necessary skills.This is certainly illustrated in the comments from students made in Section 4.2.Interestingly, around half the students whose work was assessed referred to using Google Scholar prior to training, stating that this tool was their default search tool.After training, students indicated that they would move away from relying on Google Scholar.It should be noted that Google Scholar was included in the training, with an activity using it in the citation searching session.However, as part of the face-to-face session students compared outputs from different tools and subsequently many suggested that they would not use it in future.Many students also welcomed the skills training in applying limitations to database searching and felt this impacted considerably on the amount of time a search took.They also identified citation searching as a particularly valuable tool and one they had been completely unaware of prior to the training.The least well-received session was the 'Keeping up to date' session.This final session appeared to be valued in principle but not sufficiently helpful in the timescale of assessment for this module.Students did, however, recognise that it could be helpful to use this when studying for their dissertation or any longer piece of work where new literature is likely to come out between the initial searches and finals submission.
There are several limitations to the current study which should be noted.Firstly, the study is not experimental in nature and therefore no pre-and post-training data is available, and the flipped sessions were not compared to a face-to-face alternative.However, the richness of the qualitative data does provide insight into the experience of students learning these skills.A future study may consider integrating both approaches where half of the students undergo traditional classroom training and half have flipped sessions.These could be assessed with a modified skills development record before and after training and a reflection piece.Secondly, we relied on students accurately reporting their use of the online material in advance, but given that those who had not done the online work were given the option of doing it during their faceto-face slot and joining the later slot instead, it is likely that the majority did complete it.Finally, a limitation of the study is the sample size.This total cohort for this module was 48 students and only 27 gave consent for their work to be used in the research, therefore the sample size is relatively small.However, recent guidelines for thematic analysis (Braun & Clarke, 2013;Fugard & Potts, 2015) have suggested that studies using participant generated text should include 10-50 participants, indicating this sample size is sufficient.

Conclusions
This study demonstrated that three flipped sessions built into a core content module could significantly enhance the skill set of students with some prior database searching experience to improve the efficiency, quality and student confidence in database searching, notably reducing reliance on single tools such as Google Scholar.Students also found these sessions helped them acquire new skills in citation searching which held immediate value.Training in keeping up to date activities was less beneficial within the module itself but students recognised the potential value for future work.Notably the positive perceptions of the skills learnt were not mirrored in a correlation between the skills development and essay assessments.However, the lack of immediate impact should not detract from the learning experience students report.
I feel I have developed my confidence and skills in the literature searching … [and] I now feel I could confidently use [Scopus and Web of Science] for citation searching."

Figure 1 :
Figure 1: Coursework scores on the three elements of module assessment

Table 1 :
An overview of the preparation work for each of the three flipped sessions Record the search strategy in a way that would enable replications by others After watching the presentation, no further preparation was required.After viewing the 16-minute presentation students should be able to:  Understand what is meant by a citation search and why we might conduct one  Conduct and interpret a citation search in Web of Science, Scopus and Google Scholar  Recognise, and understand why, different citation indices may give