Using audience response systems to enhance student engagement and learning in information literacy teaching

One of the key challenges in Information Literacy (IL) teaching in higher education is ensuring student engagement. As such, active learning approaches are encouraged in order to maximise student participation and interaction with the teaching. The use of audience response systems (ARSs) is one active learning approach which is being used increasingly in IL teaching. The purpose of this study is to investigate the effectiveness of ARSs in terms of increased engagement and student learning. Previous research has explored the use of ARSs as an active learning approach in comparison to traditional lectures, but this study aims to specifically examine the effectiveness of these tools as part of an active learning pedagogy. Most existing studies have looked at a single ARS, usually clickers. With an increase in availability and functionality of online tools, and discussions at a university level about moving to a single system which makes use of students’ own devices, this study also aims to compare the effectiveness of clickers and online ARSs. A controlled study was carried out on two cohorts of medical students at Queen Mary University of London comparing the use of clickers, online response tools, or a mixture of the two, to teaching without ARSs. Class observation and student evaluation were used to measure student engagement, and quizzes and student confidence levels to measure student learning. Results of the study showed that ARSs, when used as part of an active learning pedagogy, are an effective tool in terms of increasing student engagement, and have a generally positive impact on student learning, with online tools being slightly more effective than clickers. The study provides evidence which can be used by IL practitioners to help integrate ARSs into their teaching as well as inform institutional decisions on the use of these tools.


Introduction
It can be a challenge to engage students in information literacy (IL) teaching, and librarians need to find innovative ways to deliver teaching in order to ensure maximum engagement, which enables students to develop these key skills.At Queen Mary University of London (QMUL) librarians in the Teaching & Learning Support Team have been working on active approaches to IL teaching.One method which was being used elsewhere in the University, and has been adopted by the librarians, is the use of audience response systems (ARSs).Whilst clickers have been used by some departments for a number of years, librarians have been at the forefront of developing the use of online ARSs.Although both students and teaching staff seem to appreciate using these tools, it was felt that it would be helpful to provide some evidence to endorse their use.
The aim of this study is to measure the effectiveness of ARSs within IL teaching.Many previous studies use ARSs to examine the effectiveness of active learning, against traditional lecture based teaching (Holderied, 2011;Ross & Furno, 2011;Walker & Pearce, 2014).It has been shown that active learning approaches are effective in IL teaching (Bell, 2007;Boss, Angell, & Tewell, 2015;Hegarty, Carbery, & Hurley, 2009;Holderied, 2011;Lahlafi, Rushton, & Stretton, 2012), and so this study aims to go beyond the existing research by specifically examining the effectiveness of these tools as part of an active learning pedagogy.Initially the study aimed to address two main questions:  Does the use of ARSs increase student engagement in IL classes? Do ARSs have a positive impact on student learning in IL classes?
Most previous research on the use of ARSs in IL teaching was carried out a number of years ago and therefore focuses on the use of clickers.In the last few years, particularly with the growth in smartphone usage, online tools have become more widely available, with ever-increasing functionality.In order to build on existing research, as well as to provide data to feed into recent university wide discussions around the possibility of moving to a web-based system utilising students' own devices, it was decided to add an additional research question:  Are clickers or online ARSs, or a combination of both, most effective in impacting on student engagement and learning?
These questions were addressed by carrying out a controlled study on two cohorts of medical students at QMUL, comparing the use of clickers, online ARSs, or a mixture of both, as part of an active learning approach, to teaching without ARSs.The study should provide librarians with data and analysis which will enable them to make informed decisions as to whether ARSs are an effective tool which they might want to consider using in their own teaching to improve student engagement and learning.It could also potentially inform decisions, both at a library and institutional level, as to which ARSs might be most effective.

Active learning
One of the main challenges in IL teaching is keeping students engaged.Whilst librarians are continually working with academic departments to ensure that IL teaching is fully integrated within the curriculum (Burgoyne & Chuppa-Cornell, 2015;Clairoux, Desbiens, Clar, Dupont, & St-Jean, 2013;Kavanagh, 2011;Moore, Black, Glackin, Ruppel, & Watson, 2015;Mullins, 2014;Rae & Hunn, 2015), this is still not universally the case.This means that IL teaching is often seen as an add-on (Ross & Furno, 2011) and can be viewed by students as peripheral to their main course of study.There can also be a feeling among students in higher years that, having attended IL sessions previously, they have done it all before despite the fact that they are learning new skills, or different applications of them.Bell (2007) describes this as the "I already know this" or IAKT syndrome.Traditional lectures are ineffective in teaching IL, not incentivising engagement with material that is outside students' subject specialism and therefore not seen as particularly interesting (Walker, Finley, MacMillan, & Skarl, 2013).Didactic teaching and passive learning do not provide opportunities to develop IL skills which require practical application (Verlander & Scutt, 2009).
Librarians are always looking for ways to make IL teaching more diverse, with Ross and Furno (2011) suggesting that finding the best pedagogical practices is an iterative process that takes patience and courage to fail.There is a trend towards making the delivery of IL education an active learning experience (Bell, 2007;Boss et al., 2015;Hegarty et al., 2009;Holderied, 2011;Lahlafi et al., 2012).Active learning refers to any learning activity which involves active participation of the student (Higher Education Academy, 2008), rather than the one-way communication between teacher and student characteristic of passive teaching methods (Holderied, 2011).Active learning approaches are highlighted as being important and effective within higher education (HE) (Buitendijk, 2017), as they encourage deep learning and enable students to take responsibility for their own learning (Bell, 2007;Higher Education Academy, 2008).Active learning in IL teaching has been shown to engage students and increase motivation by enhancing interactivity, getting them involved in the classroom, combating the IAKT syndrome and stopping them from getting bored (Bell, 2007;Hoppenfeld, 2012;Jones, Peters, & Shields, 2007).Utilising a range of interactive methods helps to facilitate learning and improve understanding by ensuring that teaching is fully inclusive; accommodating a variety of learning styles and enabling involvement of those with disabilities, and others who could potentially feel excluded (Verlander & Scutt, 2009).
Active learning approaches regularly used within IL teaching include hands-on activities, group discussions, games, and the use of ARSs.

Using ARSs
The use of ARSs, sometimes referred to as student, classroom or personal response systems, or electronic voting systems, is one example of an active learning approach which has been increasingly employed in HE and in IL teaching during the last ten years.There are two main types of ARSs being used in HE: clickers and online tools.The former utilise small handsets, usually known as clickers, which are used to interact wirelessly with presentation software such as PowerPoint.Relevant software needs to be installed on the presentation computer and a dongle used for wireless connectivity.TurningPoint (http://www.turningtechnologies.co.uk), used at QMUL, and iClicker (https://www1.iclicker.com)are examples of this type of technology.Barriers to using clickers can be the cost and logistics of acquiring and distributing sufficient devices (Erjavec, 2010;Wiley, 2015), technical issues with wireless connection and channels, and the need to have the software installed on any computers used to prepare or deliver presentations.With online ARSs participants can use any internet connected device.The software is entirely cloud-based and the basic versions are generally freely available.Some commonly used online ARSs are Poll Everywhere (https://www.polleverywhere.com),Mentimeter (https://www.mentimeter.com)and Socrative (https://www.socrative.com).Problems identified with these systems include restrictions on number of participants or questions available in freely available versions, the need for all students to have access to internet connected devices, and the possibility of students getting distracted if they are encouraged to use technology in class (Keogh & Wang, 2010).
There are a number of effective uses for ARSs in IL teaching, which allow teachers and students to get real time results from the audience (Hoppenfeld, 2012).A common use is to check student knowledge, for example at the beginning of a lesson, to help ascertain the current skill level of the students, respond to any misunderstandings, and tailor teaching to the needs of the students (Burnett & Collins, 2007;Deleo, Eichenholtz, & Sosin, 2009).Educational research suggests that students like using ARSs as they can compare responses with the group and are able to check their understanding against other students (Heaslip, Donovan, & Cullen, 2014;McCartan & Peel, 2011).They can also be used as an assessment tool, usually formatively, to assess student comprehension and reinforce learning outcomes (Hoppenfeld, 2012).An advantage of using ARSs as an assessment tool is their ability to provide immediate diagnostic feedback (Davies, Mullan, & Feldman, 2017;McCartan & Peel, 2011).By highlighting areas of development to both teachers and students, this helps to close gaps between current and desired performance as advocated by Nicol and Macfarlane-Dick (2006) in their work on formative assessment.Another use of this technology is to survey students, for example to find out what resources they use (Hoppenfeld, 2012).They can also be used as an effective way to feed back from class activities, when students have been asked to find something out, or carry out a task, and then share their responses with the class (Burnett & Collins, 2007).This is supported by educational research, which suggests that their use can help to foster teamwork and collaboration, and promote peer learning (Pettit, McCoy, Kinney, & Schwartz, 2015;Waldock, 2013).Hoppenfeld (2012) suggests that audience response questions should be integrated at points throughout the session to break up the content, which is particularly important if the session is otherwise passive in nature.

Increased satisfaction
Whilst student satisfaction is not necessarily an indicator of engagement or learning, it is likely that students who enjoy learning are more likely to be engaged with the teaching and learn more.Most research on the use of ARSs within IL teaching has reported positive feedback from the students (Holderied, 2011;Hoppenfeld, 2012;Keogh & Wang, 2010;Ross & Furno, 2011).In a study at the American University of Sharjah in the United Arab Emirates, 83% of students felt that clickers were an effective learning tool (Ross & Furno, 2011).In another study, nursing students commented that they found the use of clickers helpful, particularly in terms of their visual and participatory nature (Keogh & Wang, 2010).Holderied's (2011) study with students on an English composition course reported that students' responses to questions about enjoyment were more positive in the group using clickers, and in a study at Texas A&M University students actually asked for them to be used more (Hoppenfeld, 2012).

Increased engagement
Increasing student engagement in IL teaching is one of the prime motivators for using ARSs (Holderied, 2011;Jones et al., 2007;Ross & Furno, 2011;Walker & Pearce, 2014), with this technology recognised as promoting involvement and engagement (E-Learning Unit, 2017a;Holderied, 2011).Hoppenfeld (2012) found that the use of an ARS made students more noticeably attentive, particularly during the polling questions.Several other studies have reported on the palpable buzz among students as they answer the questions (Burnett & Collins, 2007;Walker & Pearce, 2014).Using ARSs in class allows all students to participate equally and enables inclusion by giving a voice to those who do not feel able to speak in class (McCartan & Peel, 2011).Whilst one study did not find a statistically significant engagement effect in classes using clickers (Walker & Pearce, 2014), all other studies analysing the effect of ARSs on student engagement in IL teaching reported a positive impact, including increased fun (Burnett & Collins, 2007) and active participation (Keogh & Wang, 2010).Deleo et al.'s study (2009) with graduate students on an Educational Leadership and Technology program found that students using clickers were engaged in a way that the librarian had not experienced before.

Increased learning
A few studies have considered the effect of ARSs on learning within IL classes or courses (Holderied, 2011;Ross & Furno, 2011;Walker & Pearce, 2014).However, there seems to be little consensus on their impact in this area.Ross and Furno (2011) used pre-and post-tests to assess IL skills and found that there was a 6% skills improvement following attendance at a class using clickers.Holderied (2011), using pre-and post-tests to assess the achievement of learning outcomes, also reported increased success, with the clicker group performing better than the control group by an average of 4.02%.However, Walker and Pearce (2014) found no difference in achievement of learning outcomes for first year undergraduates when comparing traditional lectures and user-centered approaches including clickers.Holderied (2011) sums up the situation by concluding that while interactive technologies can promote the achievement of learning outcomes in some instances, further research is needed to ascertain their impact on effective learning within IL.

Methods
It was decided that the best way to test the effectiveness of ARSs would be to compare levels of engagement, student satisfaction, learning outcomes, and levels of confidence, across groups which would be as similar as possible apart from their use of ARSs.Two IL sessions were chosen, with two different cohorts of medical students, which are repeated multiple times: first year Netskills and third year Evidence Based Medicine (EBM).In each case there were four different interventions, with groups either using clickers, online ARSs (Mentimeter or Socrative), a mixture of both, and a control group who did not use ARSs at all.As the aim of the study was to compare different tools as part of an active learning pedagogy, in both cases these sessions involved other active learning elements, such as group discussion and hands-on activities.These remained constant, as did the content of the session, with the only variable being the use of ARSs.

Third year Evidence Based Medicine sessions
These sessions introduce EBM, how to search effectively for high quality evidence, and critical appraisal skills.The session is repeated with twelve groups of around 25 students.Three groups were assigned to each test intervention: clickers (n=79), online ARSs (n=75), mixed ARSs (n=75), and control (n=69).ARSs were used to get students to identify the highest levels of evidence, through multiple choice questions using clickers, or sliding scales on Mentimeter, and also to identify key elements to consider when critically appraising a Randomised Controlled Trial.In the control groups students were asked to shout out answers.
Students then worked in small groups to critically appraise an article, answering a questionnaire mainly with yes/no/can't tell answers.Clickers or Socrative were then used to collect responses and feed back to the class in the intervention groups.Feedback from the control groups was collected verbally by getting one or two groups to share their answers to each question.

First year Netskills sessions
These sessions introduce the students to library resources and basic IL skills.There were four groups with around 65 students in each, so each group was assigned to one of the interventions: clickers (n=65), online ARSs (n=64), mixed ARSs (n=63), and control (n=65).ARSs were used to ask some basic questions to get to know the students for example, 'have you visited the library yet?'.In the control group students were asked to raise their hands.Clickers or Mentimeter were used to ask students to identify elements of a citation and abstract.The control group were talked through the different elements by the teacher.ARSs were also used to get students to think of keywords to describe a photograph, with the control group asked to shout out responses.

Data collection and analysis
Measuring student engagement in class can be difficult.Library staff leading and assisting with the sessions observed the way the students participated in the class, how involved they got with the activities, and how engaged they seemed to be overall.Although observation can be subjective, it was felt that this was the most effective way to gauge levels of interaction, participation and engagement within the class.In addition to this students were asked to complete the standard evaluation form (Appendix A), which is used for all IL sessions at QMUL.Whilst not necessarily indicating increased engagement, it was felt that student satisfaction is likely to have an impact on both engagement and learning.Students are asked to rank a number of components on a scale from strongly agree to strongly disagree.A number of these elements could be influenced by the use of ARSs, including whether the session addressed their needs, clarity of presentation, organisation of the session, and overall experience.The total agree and strongly agree responses for all components were added together to give an indication of overall satisfaction.Although the question about venue was unlikely to be influenced by the use of ARSs, it was felt that including this would not have an impact on the results, as all sessions with each cohort took place in the same room.Free text comments offered as part of the student evaluation were also analysed, as it was felt that these might provide an insight into levels of engagement.All quantitative data from the evaluation forms was entered into an Excel spreadsheet for manipulation and analysis.Qualitative data was searched from within the Access database used to record student feedback from IL sessions at QMUL, to identify free text comments pertaining to the use of ARSs.
In order to measure the impact of ARSs on student learning, the participants were also asked to complete a short quiz on QMplus, the university's virtual learning environment, at the end of the session (Appendices B and C).The aim of this was to test the learning outcomes and analyse whether there was any difference in how much students had learnt between the different groups.
Results of the quizzes are stored automatically on QMplus, and from here they were sorted by group and exported into Excel for data manipulation and analysis.The questions asked on the evaluation form (Appendix A), about levels of confidence before and after the session, were also considered to be a useful measure of students' perceptions of their learning.Confidence was taken as those strongly agreeing or agreeing that they were confident about the content of the session.
The percentage difference between those confident before the session, and afterwards, was calculated, with the data gathered and analysed in Excel.

Student engagement
Evaluating student engagement in a meaningful way can be challenging, but the teachers and library staff assisting with the sessions observed that in all cases the students using the ARSs were more engaged with the teaching.For the first years in particular, the use of clickers or Mentimeter created a perceptible buzz of excitement (Burnett & Collins, 2007;Walker & Pearce, 2014), with the students discussing their responses with those around them.With the third years those using the ARSs seemed to be more focused on the task in hand, particularly during the group critical appraisal.They all provided responses to all the questions, whereas getting verbal feedback from the groups not using ARSs was very difficult.The vibe in the sessions also seemed to be much more positive, with a much more collaborative atmosphere and productive conversations taking place.

Student evaluation
At the end of the session students were asked to complete the standard IL session evaluation form (Appendix A).Students ranked a number of components from strongly agree to strongly disagree and the total responses for all components were added together.The total number of strongly agrees or agrees for both cohorts slightly favoured the use of one or more ARSs, but only by a small amount.

ARSs at QMUL
At QMUL such tools are encouraged as a way to promote engagement, and the use of TurningPoint clickers has become widespread (E-Learning Unit, 2017a).Some academic departments issue clickers to students for the duration of their course, whereas others, including the School of Medicine & Dentistry, have a number of clickers available for staff to use in teaching when desired.The Library also has a number of clickers for use with small groups, and others can be borrowed from the University's E-Learning Unit.There has recently been much greater interest in the use of online ARSs due to issues of lack of availability of clickers, and not having the appropriate software installed on teaching computers.As a result there are currently discussions ongoing across the university about the possibility of an institutional adoption of an ARS which uses students' own devices (E-Learning Unit, 2017b).It is hoped that this study, along with examples of the use of ARSs within the Library, will feed into these discussions.For the first years the online ARS group had the most positive feedback, with 96.4% of the responses being either strongly agree or agree.This was followed by the clicker group (94.8%), then the mixed ARS group (94%), but the control group was only slightly behind with 93.7%.The difference was greater when just taking into account strongly agree responses, between 44.6% and 47.6% in the test groups compared to only 33.1% in the control group.
Feedback from the third years again slightly favoured the use of ARSs, but in this case the mixed ARS group was most positive with 92.7% of the responses being either strongly agree or agree, followed by the online ARS group (92.6%), the clickers group (92%) and the control group again at the bottom with 91.1%.In this case, just taking strongly agree responses into account did not favour ARSs, with the control group coming out ahead of the online group.Overall these results suggest that the use of ARSs have a small positive impact on student satisfaction with IL teaching, more pronounced among first years, with online ARSs receiving slightly better feedback than clickers.
Free text comments on the session evaluation demonstrated that the students in all groups found the use of ARSs helped make the sessions more interactive and enjoyable.First year students commented that 'the voting system was really unique' and that they particularly liked the 'cool clicky response thing'.A third year student hinted at increased engagement by commenting that 'the interactive quizzes kept me paying attention'.

Quiz to test learning outcomes
Both groups were asked to do a short quiz on QMplus at the end of the session (Appendices B and C) in order to test how well the students met the learning outcomes.In the first year groups those using ARSs in the sessions performed better in the quiz, with the control group achieving an average of 65%, the mixed ARS group 66.2%, clickers group 67.7% and online group 68.5%.The average score across the ARS groups was 67.5%.The third year quiz revealed very close results with only 1.3% separating the four groups.The online group again came out on top with an average of 77.9%.However, the mixed ARS group came out bottom with an average of 76.3% Strongly agree Agree compared to the average in the control group of 76.8%.Looking at the results of each individual group showed that one of the mixed ARS groups had results that were much lower than any other group and brought down the average by more than 2%.

Levels of confidence
On the session evaluation forms (Appendix A) students are asked to indicate whether they are confident about the session content prior to the session and after the session.The percentage difference in those strongly agreeing or agreeing that they were confident before and after the session was calculated as a good indicator of how much students thought they had learnt during the session.For both the first and third years the increase in confidence was significantly greater in the groups using some form of ARS.For the first years the percentage difference in those reporting that they were confident before the session and after was 40% 42.9% in the online ARS group, and 51.7% in the mixed ARS group.This seems to suggest that the students feel they are learning more when they are using ARSs in the sessions.

Discussion
While it is extremely difficult to accurately measure student engagement, this study shows that, generally, the use of ARSs in IL teaching has a positive impact.The atmosphere in all classes using the ARSs seemed to be positive, with students appearing to be more interested in the activities, interacting with the teachers and one another, participating fully, and generally enjoying the session more, which reflects the findings of previous studies (Burnett & Collins, 2007;Deleo et al., 2009;Keogh & Wang, 2010).This was particularly true in the sessions with the third years, where the IAKT syndrome (Bell, 2007) had previously been a problem, making it very difficult to engage the students in the activities.During the group critical appraisal, students not using the ARSs seemed to get easily sidetracked.It also proved very difficult to get them to feed back their responses to the group without having to pick on people, which can be uncomfortable both for the teachers and students.Using the ARSs not only made gathering feedback easier, but students seemed to engage much more with the group discussion and keep to the task at hand.This may be because they knew that their responses would be recorded and shared with the rest of the class.This encouraged a much more collaborative approach, where students actively discussed the article and could learn from each other; positive aspects of using ARSs previously identified in educational literature (Pettit et al., 2015;Waldock, 2013).
The increase in engagement is reflected by the responses gathered from the student evaluation.
As in previous studies (Holderied, 2011;Hoppenfeld, 2012;Keogh & Wang, 2010;Ross & Furno, 2011) the feedback from the students was positive, although the responses from the groups using ARSs was only slightly more positive than the control group.Unlike most previous studies, which compared the use of ARSs to traditional lectures (Holderied, 2011;Ross & Furno, 2011;Walker & Pearce, 2014), the classes in this study used a number of other active learning approaches, such as group discussion, hands-on activities, and games.It is therefore perhaps to be expected that the increase in satisfaction levels solely due to the use of ARSs is likely to be relatively small.However, student satisfaction was greater among the first years, which could perhaps be because younger students appreciate the fun and interactive elements more.Although high levels of satisfaction do not necessarily demonstrate high levels of student engagement, this is very likely to be the case, as is indicated in this study.
In terms of learning, the quiz results for the first years showed a positive correlation between use of ARSs and learning outcomes.However, this was not the case with the third year groups.This mirrors the results of previous studies, some of which reported a positive impact on learning outcomes (Holderied, 2011;Ross & Furno, 2011), with others finding no clear benefit (Walker & Pearce, 2014).Potentially it may be that active learning methods are more beneficial for first year students than older students, and in fact much of the research on the use of ARSs tends to be focused on first years.However, Walker and Pearce's study (2014), which did not find the clicker teaching effective, was also carried out with first year students.Both cohorts in the current study demonstrated a greater increase in confidence levels among the groups using the ARSs.This suggests that the students think they have learnt more, even if this is not backed up by the quiz results.
The groups of third year students using a combination of clickers and online response tools, threw up some unusual results.The results of the quiz suggested that this group had learnt the least from the teaching.This was mainly due to very poor results in one of the three groups, for which there was no obvious explanation.However, the student satisfaction and increase in confidence was highest in the mixed ARS groups.Although there is no reason why the groups should not be similar in terms of prior knowledge, this suggests that knowledge levels could have been lower at the start of the session in the mixed ARS groups, which corresponds to the differences in confidence levels before the session.Whilst these students were happy with the teaching, and felt that they learnt a lot, this did not bring them up to the level of the other groups.This suggests that a pre-and post-test would have been useful to better measure student learning.
The use of ARSs is very helpful in terms of increasing the inclusivity of IL teaching.Students are often nervous to answer questions in class, for fear of losing face if they get something wrong, which is a particular issue in certain cultures (Huang, Davison, & Gu, 2008).ARSs allow all students to answer the questions anonymously, which is one of the aspects that students really appreciate (Heaslip et al., 2014).It also allows students to easily share ideas, allowing those who do not find it easy to speak up in class to fully participate.From observing the classes using ARSs, all students were participating with the activities, rather than just a few students who put up their hands or were happy to share their answers orally in front of their peers.Students' reaction to the visual representation of results was very positive, with word clouds in Mentimeter in particular appearing to cause a level of excitement among the students.These visual representations are particularly beneficial to those with visual learning styles or specific learning difficulties (Association for Higher Education Access & Disability, 2017).
As well as demonstrating the benefits of using ARSs as part of an active learning pedagogy within IL teaching, this study also set out to identify any differences in effectiveness between different types of tools.This goes beyond most of the existing literature, which examined just one type of ARS, mostly clickers, usually in comparison to traditional lecture formats (Holderied, 2011;Ross & Furno, 2011;Walker & Pearce, 2014).In terms of the learning outcomes, demonstrated by the quiz results, the groups using online ARSs came out top, followed by the clicker groups, in both cohorts.This is corroborated by the increase in confidence levels among the first years.The mixed ARS groups showed the largest confidence increase among the third year students, but again the online ARS groups were more confident than the clicker groups.This could potentially suggest that online tools are more helpful in promoting student learning than clickers.In terms of student feedback, the online ARS groups showed higher levels of satisfaction than the clicker groups, but again the mixed ARS groups came out top with the third years, but not so well with the first years.It is unclear why the first year groups did not respond so well to using a combination of online response tools and clickers.It could be that the sessions lacked consistency, or potentially the students felt a bit overwhelmed by too many different technologies.
This study shows online ARSs to be more effective than clickers in terms of both engagement and learning.There are a number of potential reasons for this.The visual impact tends to be more dynamic, which seems to appeal to the students.Also, a system like Mentimeter has a very large range of questions types, whereas clickers, at least those used at QMUL, tend to be restricted more to multiple choice questions, or similar.Some question types perhaps work better for certain activities, making them more interesting and useful for the students, and also allow for more variety.From the teaching point of view, online ARSs tend to be very easy to use and do not require specialised hardware or software.However, there are some drawbacks with these tools, notably the limits imposed on free versions, and the reliance on all students having access to an internet-connected device.

Limitations
Whilst the content of the sessions was identical, apart from the use of ARSs, the format of the questions varied slightly due to the differing nature of the tools used.For example, a sliding scale question used in Mentimeter instead of a multiple choice question using clickers.However, as this study was aiming to compare ARSs, the enhanced functionality of some of the tools could potentially be an important factor in measuring their effectiveness.
The QMplus quiz at the end of the class was an effective tool in measuring students' knowledge, but confidence levels reported prior to the class suggest that there might have been a variation in skill level between groups at the outset.Unfortunately, due to the way the timetables are organised, it would not have been possible to allocate the students to groups randomly, which might have minimised other confounding factors.Using pre-as well as post-tests would perhaps have enabled a more accurate measurement of student learning, as this would have taken into account existing skill levels.However, as these classes both take place at the beginning of the academic year, it would be difficult to get the students to take the quiz prior to the session, and doing it at the beginning of the class would have taken valuable teaching time.
Although the quiz measures the students' knowledge directly after the class, this study did not look at long-term impact on learning.It would be interesting to test students again later in their course to see whether the use of ARSs makes any difference to how well the information is retained, or if there is any difference in whether students are using the skills learnt.Future work on this would prove interesting.
This study was carried out in an HE institution and the results might not be generalisable to other settings.It also focused on one discipline, so it would be interesting to assess whether the results would be any different for students from different disciplines.

Conclusion
The results of the study have shown a positive impact of using ARSs as part of an active learning pedagogy.This adds to previous research which had shown that ARSs were effective as an active learning tool compared to passive teaching methods.Observations in class suggested that the students were more engaged in the groups using ARSs, which was supported by increased levels of satisfaction.
The impact on student learning was not so clear cut.In both cohorts there was a greater increase in confidence levels in the groups using ARSs, which would suggest that the students in those groups certainly felt that they learnt more.This was backed up by the results of the end of session quiz taken by the first years, in which the control group performed least well.However, the results of the third year quiz did not reflect this pattern.It can potentially be deduced that the use of ARSs is effective on the learning of first year students.Overall, whilst the study shows that students clearly feel that ARSs have a positive impact on their learning, the true impact of this has not been shown conclusively in this study.
In contrast to previous research, this study also aimed to examine whether there was any difference in effectiveness between clickers and online tools.Whilst there was no perceptible difference in engagement levels in class, the satisfaction levels were higher for the groups using online tools than clickers.The same pattern was also seen in terms of student learning, with both increase in confidence levels and quiz results favouring online tools.This suggests that online ARSs are more effective than clickers in improving both student engagement and learning.No firm conclusions can be made about whether using a combination of tools is more or less effective than using just one type.
By demonstrating a positive impact on student engagement, and to a lesser extent student learning, it is hoped that the results of this study will provide justification to other IL practitioners using or considering the use of ARSs in their teaching.It also provides some information to assist with decisions about which tools to use, or potentially purchase, at a library or institutional level.

Figure 1 :
Figure 1: Session evaluation responses for first year groups

Figure 2 :
Figure 2: Session evaluation responses for third year groups

Figure 3 :Figure 4 :
Figure 3: Quiz results for first year groups

Figure 5 :Figure 6 :
Figure 5: Percentage of first year students strongly agreeing or agreeing that they were confident about the content of the session in the control group, 46.2% in the mixed ARS group, 48.8% in the clickers group, and 57.2% in the online ARS group.For the third years the percentage difference in the control group was 34.4%, with 38.4% in the clicker group,