How can you tell if it’s working? Recent developments in impact evaluation and their implications for information literacy practice.

This paper surveys the (patchy and uneven) advances in LIS impact evaluation over the past ten years and notes the surge forward in public library impact evaluation, before looking more broadly at the international and educational impact evaluation scene and noting the advance of programme-theory driven approaches. The authors then identify various trends drawn from the wider evaluation discourse that they think are likely to be relevant to Information literacy (IL) practitioners, academic staff, employers and others who are concerned with impact evaluation of IL work. The trends identified are: Implications for IL practitioners are in relation to each of The authors predict that over the next ten years there will be a strong focus on whether IL interventions are having an impact in combating misinformation and disinformation; more systematic and sustained approaches to IL impact evaluation in the health and higher education sectors but less so in some school libraries and other settings. They think that the more proactive public libraries will adopt IL evaluation approaches, that workplace IL will continue to depend upon the organisational culture, and that research on information seeking in context will shed light on evaluation priorities. Finally, they hope that future IL work will be underpinned by programme theory-based evaluation.


Abstract
This paper surveys the (patchy and uneven) advances in LIS impact evaluation over the past ten years and notes the surge forward in public library impact evaluation, before looking more broadly at the international and educational impact evaluation scene and noting the advance of programme-theory driven approaches. The authors then identify various trends drawn from the wider evaluation discourse that they think are likely to be relevant to Information literacy (IL) practitioners, academic staff, employers and others who are concerned with impact evaluation of IL work.

The trends identified are:
 growing clarity about the levels of evaluation expertise needed to deliver information literacy support from the perspectives of leaders of LIS education programmes, staff of academic institutions, library leaders and managers and IL practitioners,  growing interest in more inclusive or democratic approaches to impact evaluation  the limitations of the simple logic model of evaluation  re-purposing of existing data to meet new evaluation needs  collecting and presenting stories of change as impact evaluation evidence. Implications for IL practitioners are offered in relation to each of these trends.
The authors predict that over the next ten years there will be a strong focus on whether IL interventions are having an impact in combating misinformation and disinformation; more systematic and sustained approaches to IL impact evaluation in the health and higher education sectors but less so in some school libraries and other settings. They think that the more proactive public libraries will adopt IL evaluation approaches, that workplace IL will continue to depend upon the organisational culture, and that research on information seeking in context will shed light on evaluation priorities. Finally, they hope that future IL work will be underpinned by programme theory-based evaluation.
Keywords impact evaluation; information literacy; programme theory; education programmes; performance measurement; evidence

Advances in library impact evaluation
Throughout this paper, we take the term 'impact evaluation' to mean evaluating any effect of the service (or of an event or initiative) that contributes to change to an individual, group or community.
We began our engagement with impact evaluation in 1991 when the then British Library Research and Development Department commissioned us to conduct a UK national research study of the effective school library (Streatfield & Markless, 1994). Our first problem was to decide what we meant by 'effective' and we chose to focus on the contribution of school libraries to teaching and learning in the school. As soon as we made that decision we were irrevocably committed to a move away from researching what library staff did (which was the norm at that time) towards seeing whether and how the students and the teaching staff in the chosen schools were affected by their interactions with the library and its staff. We have been focusing on impact evaluation in a wide variety of library settings ever since.
Before homing in on Information literacy (IL), we think it will be useful to look at the notable advances in library impact evaluation more generally over the past ten years, since the publication of the first issue of this journal. We will then point to several developments in international development and education evaluation which we think have implications for IL practitioners.
In that decade the increased emphasis on accountability, value for money and evidence-based practice has made advances in impact evaluation inevitable. However, these advances have been patchy and uneven in their application; many have been sector-specific, others have been confined to a particular aspect of library service provision, most have been largely or wholly confined to the 'developed world' and none have spanned all sectors. More specifically, for example, health librarians in North America, parts of Western Europe and in Australasia are increasingly becoming more strategic in their approach to evaluation by working to incorporate impact evaluation within evidence based practice (for example, in current library workforce development planning in the UK).
Meanwhile, academic library staff, especially in the 'developed world', are incorporating some impact evaluation of their services alongside their performance measurement activity, while academic and school libraries, particularly in the US and Scandinavia are increasingly beneficiaries of academic research work which includes some impact evaluation as well as return on investment studies (much of this work is recorded by Poll, 2016).
Perhaps the biggest recent change, and one which bucks the trend of the 'developed world' taking the lead, is in the public library field. Here, impact evaluation has moved up the policy agenda largely due to the philanthropic efforts of the Bill and Melinda Gates Foundation through their Global Libraries initiative (for example, Al, et al., 2015) and interventions by various nongovernmental organisations. Some examples of these interventions are the survey conducted for the EIFL Public Library Innovation Programme by the market survey company TNS (TNS RMS East Africa, 2011) as well as the work of Electronic Information for Libraries (2014) and Beyond Access (2015). As part of the Global Libraries programme, impact specialists in a range of countries from Africa to Eastern Europe and from South East Asia to Central and South America have developed a repertoire of evaluation interventions (for example, Streatfield, et al., 2015).
Looking at the library scene more holistically, the extent to which impact evaluation is adopted is still likely to depend on external factors (such as donor insistence), a favourable organisational environment (such as in the UK National Health Service), or an immediate pressing need (such http://dx.doi.org/10. 11645/11.1.2201 as making a case for library service survival). Individual library and information services (LIS) may also choose to focus on the impact of some or all of their activities as an important step in service development. However, when LIS are short-staffed and attempting to meet a wide range of service demands with limited and often dwindling resources, they are only likely to focus on service impact if the competition for resources with other parts of the organisation becomes acute. The library evaluation situation in many countries is much less rosy than the examples above might indicate. Any discussion of the possibilities for advances in the library impact evaluation arena should be prefaced with the question, 'Advances for whom?' A connected change over this periodwe hesitate to call it an advance is that many libraries of all kinds and their national bodies or their agents (such as The Society of College, National and University Libraries (SCONUL) and, for public libraries, the Chartered Institute for Public Finance and Accountancy, both in the UK) are collecting ever more performance and usage data, most of it quantitative. It can be difficult to make sense of these data: how do the pieces of information or data relate to each other? What is their role in enabling a service to become more effective in meeting its goals? If we are to achieve a sustained focus on impact, we need to clearly articulate the desired outcomes (what do we want to achieve and for whom) and how these will be achieved (the key supporting processes and activities). Only then can data be collected and used in relation to a coherent framework, through which the evidence can be pulled together to be more meaningful and illuminate progress towards desired aims.

Advances in evaluation
Looking beyond libraries at the broader evaluation arena over a similar time period, there has been a marked shift in the nature of the discourse, especially in the international development and education evaluation fields, which might be summarised as 'attribution or contribution?'. There has been growing disenchantment with attribution studies as the 'golden bullet' for impact evaluation in complex environments such as education and community development, or in relation to complex interventions such as those required when dealing with long-term health conditions.
Attribution studies attempt to answer the question: 'Did the programme or intervention cause the observed outcomes?' By contrast, most library impact evaluation recognises the complexity of the information worlds of library users and the range of contributory factors in any change. The core question becomes: 'Did the programme or intervention contribute to or help to bring about the observed outcomes? Attribution implies causation and involves drawing direct causal links between observed changes and specific interventions. Some questions addressing attribution might be:  Are the outcomes of interest attributable to the program?  Are the outcomes of interest changing as a result of the program?  Did the program cause the outcome of interest? For comparison, some questions related to contribution are:  Is the program contributing to the outcomes of interest?  Are the outcomes of interest changing?  Is there evidence that the program helped to achieve or was part of what caused the outcomes of interest? (Almquist, 2011) This disenchantment with attribution studies, combined with recognition of the limitations of only claiming limited contribution, has led evaluators to engage in extensive debate, both in print (for example, Rogers & Weiss, 2007;Mayne, 2012;Befani, Barnett and Stern, 2014)  Rogers and her colleagues, programme theory makes 'explicit the underlying assumptions about how programs are expected to work' and, specifically, it is a theory or model 'showing a series of intermediate outcomes or mechanisms by which by which programme activities are understood to lead to desired ends' (Rogers et al., 2000, pp.5-6). Theory based evaluation is an approach which uses this causal model to guide the evaluation. In some situations the programme theory might be appropriately represented as a single, linear causal path (simple logic model). However for many human service interventions such as in education, this is not appropriate because it does not represent either the complexity of the intervention or the complexity of the environment in which it is operating. The simple logic model leaves out other factors that might contribute to observed outcomes, including the implementation context; it assumes a stable environment and predictable outcomes.

Applying evaluation developments in IL work
In this paper, we focus on what we see as significant recent changes in thinking about evaluationwhich could be summarised as a gradual move in the library evaluation field towards deeper understanding of the nature and significance of evaluation. This quest could be more flippantly described as moving beyond collecting and analysing what we refer to as 'busyness statistics' (service process and output indicators) towards engagement with user experience of libraries. We will now identify various trends drawn from the wider evaluation discourse that we think are likely to be relevant to IL practitioners, academic staff, employers and others who are concerned with impact evaluation of IL work.
What do we see as changing or having changed since the publication of the first issue of this journal?

Levels of expertise required
What are the levels of evaluation expertise needed to deliver effective IL support? It is easy to argue that more effective evaluation of the effects of library-initiated interventions on service recipients is needed if services are to more closely meet user needs but this kind of shift presents challenges. Where can library leaders find the time and resources for more effective impact evaluation and where can they turn to secure the evaluation skills and competence required to move beyond the sorts of workshop or course evaluation that IL practitioners habitually conduct? Here again, practice varies widely across different countries and LIS sectors. For example, public libraries in several East European countries employ methodologists whose role includes guiding the evaluation efforts of their libraries (but usually focusing on performance measurement rather than impact evaluation) and various higher education (HE) institutions in the US employ assessment librarians whose role includes evaluating impact.
A notable exception here is the Impact Group (formed largely of volunteer methodologists) who are engaging in impact evaluation in Romania. Despite these examples, we are not sure how widely the need for evaluation expertise is recognised. Even in countries where LIS professionals show that they take skills development seriously by publishing skills frameworks, evaluation skills may not have high visibility. It is ironic that there has been a sustained focus on advocacy, which appears in most recent lists of required skills of librarians, but the evaluation skills required to generate local evidence for advocacy is often missing.
Part of the answer to the resource question probably lies in encouraging everyone involved to engage more fully with impact evaluation and we have some anecdotal evidence that this is beginning to happen. However, what such engagement means is likely to vary at different organisational levels. The leaders of LIS education programmes, staff in academic institutions, library leaders and managers, and IL practitioners are likely to perceive and deploy evaluation differently. http://dx.doi.org/10.11645/11.1.2201

Leaders of LIS education programmes
If managers of education programmes for future LIS staff are monitoring current practice, this may lead them to take more interest in impact evaluation of library users than their predecessors. Whether or not this is so, do their programmes equip the next generation of LIS practitioners with the skills and methods needed to evaluate their IL training? This question is worth asking because the focus of IL evaluationi.e. learner change in an educational or work settingis different from that for most LIS provision, which includes a range of possible changes to people's lives, covering but not confined to health, education, work, leisure and community engagement. (The Global Libraries Initiative identified seven main areas in which people's lives can be changed through free access to the internet (Paley et al., 2015) and that is of course only one aspect of library service provision).
If the challenge of preparing the next generation of LIS students is accepted, who should ensure that they are trained in the evaluation of IL provision in its many forms? Looking beyond education programmes, what part if any should academic staff play in supporting effective IL evaluation in practice, given that IL leaders are likely to look for help in this area? Should LIS researchers become impact evaluators with a special interest in IL?

Staff of academic institutions
One issue that needs to be explored further in the education world is how the IL training or course provision on offer is viewed by the school, college or university and how their staffs view the engagement by their library staff in providing and supporting IL development. If such provision is seen as an integral part of the education on offer, the academic leaders should be taking the lead in encouraging impact evaluation focused on students who are deemed part of any IL programme. If, on the other hand, IL work is seen as 'the librarian's curriculum', or as an inherent part of the library support for students, the library managers will need to initiate any collaboration to support impact evaluation and to navigate through research ethics issues about access to students for research purposes.

Library leaders and managers
We know from years of running workshops on impact evaluation for library leaders that most participants have a bias towards actionthey find it much easier to focus on organising activities and providing services than to clearly articulate what changes they are trying to bring about and how (that is, producing meaningful impact objectives)and to keep these at the forefront of their thinking. This is problematic, because stating clear objectives is a vital step towards evaluating impactif you are not sure what you are trying to achieve how can you tell whether you are succeeding?
We are still a long way from a situation where library leaders habitually recognise that a coherent impact evaluation strategy provides the ideal managerial tool for driving organisational change. However, compared with twenty years ago when we began running impact evaluation workshops for library leaders, it is evident from participant comments at the beginning of the workshops, that there is now more awareness of the importance of gathering evidence of the impact of library programmes. (For a summary of reasons why participants attend workshops see Markless & Streatfield, 2013, pp.20-21.)

Information literacy practitioners
Since information literacy development is inherently educational, focusing on the impact of IL education or training should in theory be less of a stretch for practitioners in the school, further and higher education sectors than for their colleagues in other LIS settings (such as public and special libraries). Staff in these sectors are likely to be more attuned to educational evaluation principles and approaches, and in addition advice and help should be available from teaching colleagues. However, undertaking performance measurement and evaluation focused on information literacy requires a range of skills which become more complex as we move beyond 'busyness statistics' towards impact evaluation. IL leaders needs to concentrate on what changes to people's activities and behaviour they want to help bring about and how these can be achieved, which requires an understanding of the pedagogic principles involved.
If a serious attempt is to be made to evaluate the effectiveness of IL efforts in educational settings, greater understanding of pedagogy is still not enough. Finding out whether students have enhanced their understanding and are applying what they have learnt involves more than looking at lesson plans and learning objectives, on the one hand, and analysing post-event reactionnaires on the other (even though these are still the most common approaches to evaluation of IL educational efforts). People involved in providing IL support in educational settings should look for ways of following up on students (and, ideally, former students) who have been exposed to their work. This can be done by, for example, conducting follow-up interviews or longitudinal studies to find out whether the IL work has contributed to changes in those involved. It is also desirable that IL providers should be reflective practitioners. But what proportion of their energies should be employed in collecting evidence about the longer-term impacts of their work?
The issue of proportionate time commitment to evaluation is even more problematic for professionally qualified IL practitioners in other settings (ranging from some schools to law firms, where IL work is only a small part of the remit). What can you reasonably do to look at the impact of your role? We suggest that you should be able to begin to find out what is working well to achieve some change, or what you need to alter, by arranging one or two focus groups for participants in your IL initiatives (ideally conducted by someone not directly involved in the IL provision). These focus groups should take what participants find useful, what specific differences the IL interventions have made to them, and what they would like to be covered differently as the starting points. Alternatively, you could ask some 'critical incident' questions, online or face-to-face. This can be done by asking respondents to identify a recent occasion in which they used specific IL strategies or materials you provided, and to describe the results.
Even the most committed IL developer will only have limited time and energy to devote to evaluating the full impact of their work, and realistically they are not likely to have the research expertise needed to conduct a comprehensive evaluation programme. Does this mean that such evaluations will only be undertaken as part of an externally funded development programme, and then only if the donor requires this? The answer at present is probably yes, and one of the most notable features of library impact evaluation in recent years has been the growing role of philanthropic bodies in encouraging LIS impact work (as outlined above in section 1). However, it should be possible for organisations to collaborate in conducting effective evaluations drawing upon this recent experience.

What are the implications for the IL community?
If IL provision is important, it is equally important where feasible to tell the story of change brought about through such work. To do this on any scale in education settings will require collaboration between teachers and LIS leaders to secure funding and to interest professional evaluators where required. More limited pooling of efforts between LIS staff and teachers has been achieved in many cases where there is a common interest in supporting student IL development.
The collaboration challenge is yet more acute for IL practitioners in other settings, although in some countries there may be opportunities to persuade the leaders of LIS education programmes to encourage students to undertake assignments focused on IL evaluation in their own setting. http://dx.doi.org/10.11645/11.1.2201 Library Schools and Library Associations should in turn think about how to develop a wider range of evaluation skills amongst their library communities. LIS professionals can also take a lead when putting together information literacy projects (such as developing online resources or running a series of workshops for paraprofessionals),by ensuring that evaluation is built in and realistically costed.

Inclusive approaches to impact evaluation
Libraries are getting better at engaging with their communities in deciding what services to develop and how to develop them, but evaluation still tends to be controlled by library managers (or, for large projects, by professional evaluators). However, we are beginning to detect some movement away from treating library users as 'evaluation subjects' and towards more inclusive evaluation. The international development evaluation literature has advanced various ideas about a more inclusive and democratic evaluation embracing a wider range of stakeholders, including marginalised groups, and involving them in designing the evaluations and in interpreting the findings (for example, Mertens, 2003;Greene, 2006;Patton, 2012). Central here is the idea of preventing qualitative evaluation from becoming just another way of enforcing the existing power relationships between organisations and their people. In library evaluation this idea takes various forms, ranging from consultation with users about the impact areas for evaluations and forming evaluation partnerships with stakeholders to establishing panels of library staff and users to review impact evidence and take editorial decisions about the presentation of results. These three principles appear to present an appropriate challenge for IL evaluators.

The simple logic model and its limitations
As library services become more innovative, the limitations of simple evaluation models are beginning to show. The usual approach to library evaluation is sometimes called the simple logic model by evaluators. It assumes that there is a series of direct relationships between service inputs, processes, outputs and outcomes (expressed as quantified outputs). However, as libraries increasingly seek to take advantage of ICT developments by introducing new services, design different spaces to engage their community in new activities, and venture into teaching and learning in the area of information literacy, they are moving into the area of 'emergent evaluation' (sometimes described as 'developmental evaluation ' -Patton, 2011) in which it is not straightforward to predict where change will appear and therefore, where to focus the evaluation. Rogers (2008) explores a range of evaluative models. At her two extremes are the simple logic model and the complex logic model, where the paths from action to impact are complex, with disproportionate relationships (in which, at critical levels, a small change can make a big difference) and emergent impacts (which cannot readily be specified at the outset). The more advanced library services, including information literacy interventions, appear to have all the characteristics of the complex logic model, but so far we have seen more discussion than action in addressing the issue.
The evaluation discourse in Europe, as exemplified by the European Evaluation Society conference in Maastricht (September, 2016) increasingly recognises that complexity is the norm and that complex situations and initiatives require flexible, agile evaluation approaches to deal with multiple factors, relationships and layers. By complexity we mean a system in which relationships are non-linear (as noted above); there are multiple perspectives to encompass, the system is dynamic and produces unpredictable change. This is 'messy space': everything is connected because incidents or changes in one part of the system affect all other parts (Preskill & Gopal, 2014).
In responding to this complexity, evaluators give central importance to rigorous and systematic early articulation of the ways in which the programme expects to bring about clearly identified changes in individuals and communities at a range of different levels. The usual approach is to build a Theory of Change to encompass these elements, as described in the next section.

What are the implications for IL practitioners?
We feel that the key missing element in evaluating LIS work is usually a strong Theory of Change (ToC). This is a description and illustration of how and why a desired change is expected to happen. It focuses in particular on mapping out or 'filling in' between what a programme or change initiative does (its activities, mechanisms or interventions) and how this leads to desired goals being achieved. This 'filling in' process aims to fill what is usually described by ToC adherents as the 'missing middle', because traditional programme plans omit this crucial 'how' element.
Building a ToC starts with formulating the desired long-term goals and then working back from these to identify all the conditions (outcomes) that must be in place (and how these interdepend on each other) for the goals to occuror the specific differences you want to make and how you expect to get there. When applied to IL work, this will lead the organisers to elucidate how learning can occur and what conditions need to be in place for this learning to happen and to be applied. These are all mapped out in an Outcomes Framework. An important element in this work is to make systematic use of relevant research literature from appropriate disciplines (in this case pedagogic and IL development) to inform the ToC.
The main elements to be taken into account in assembling a ToC for IL work will depend upon the nature of the intervention. For example, if in a school or college setting a practitioner decides to mount a series of lessons on IL, what has to happen to ensure that students will apply IL learning when performing assignments? To answer this question calls for understanding of the learning processes involved; organisation of ongoing feedback, peer support and monitoring; and provision of resources in mixed formats that can be accessed by everyone. Similar considerations will need to be addressed if there are various elements to the IL work that is under consideration (for example, if you decide that mounting an IL course or a series of workshops will not be enough to bring about the level and type of change that you are aiming to achieve).
The ToC should also reflect the context-sensitive nature of IL provision: for example, law librarians working with Partners will need a different ToC from school librarians working with primary classes, and this is likely to be different again from a ToC where students are learning IL skills and strategies online. http://dx.doi.org/10.11645/11.1.2201 One effect of creating a ToC is to bring into question our traditional methods of data collection for evaluating training or education programmes and to encourage evaluators to use mixed methods (Greene, 2008). This is important when the evaluation norm is usually to collect lots of data but without being able to show whether and how the IL work contributes to the change being sought. Abandoning the simple logic model means moving away from the generation of data that project educational change as a simple linear process, and adopting instead an approach that recognises the complexity of evaluating change. Additionally, this move makes it more natural to think about contribution analysis (rather than unrealistic attribution claims) that looks at causal bundles which together trigger a change, and process tracing (to see what happens at each of the interacting points in the process). Both these terms are borrowed from the world of programme theory, which also gave rise to the Theory of Change concept.
We believe that if library and information researchers adopt and apply these principles in their evaluation work this will make it easier for library leaders and IL interventions to be seen as credible. This is because the library leaders and IL managers will less often be required to make simplistic and unconvincing attribution claims for their services. The drawback, of course, is that more credible IL evaluation requires a significant and sustained investment of time and resources. There is therefore an accompanying need for library leaders to be able to express a sound strategic vision and be willing to undertake difficult conversations with senior university staff.

Re-purposing of existing data
We have suggested that adherence to the simple logic model for LIS evaluation leads to the generation of much unnecessary data, but in combination with more nuanced approaches these data can be used meaningfully. For example, much LIS performance data does serve a useful performance measurement purpose in showing how efficiently library services are operating. What we sometimes call 'busy-ness statistics' are an important part of the picture, when used alongside evidence of the impact of services on library users or participants in education and training. Similarly, performance data can be used in combination with impact evidence both to guide decisions about where to focus impact evaluation efforts (pointing to strengths and weaknesses in the operation of services) and to supplement impact evidence being collected to show whether an evaluation objective is being met. For example, in educational settings qualitative and quantitative data can be combined to provide a rounded picture of how provision of IS training contributes to student retention and progression.
As more and better e-based LIS services become available the associated software accrues massive amounts of data in the process, and this begins to present possibilities for looking at the impact of IL education and training in other ways. One method of doing this, which is growing in importance, is through electronic tracking of people's information search behaviour or 'digital footsteps' (Nicholas & Rowlands, 2008). This approach, and the variations which are likely to emerge, will enable evaluators to move on from reliance on how people say they use information or libraries in their work, education and social lives to seeing how they actually use them.

What are the implications for IL practitioners?
Since we are at the beginning of what will probably become a powerful trend in non-commercial exploitation of tracking data it is difficult to forecast the implications for IL evaluation practice. However, use of such data in combination with research into student information-related behaviour, such as Heinström's (2003) work on student information searching, may well bring about insights into the impact of IL work. http://dx.doi.org/10.11645/11.1.2201

Stories of change as evaluation evidence
You can collect stories to give a much richer picture of how the services are working and how well or badly they are being received. Constructed service case studies can be built up by gathering stories of how various managers plan, set up and deliver their IL work or other services (and their view of how users respond to these services)offering a chance to illuminate the essential elements of this 'hidden world'. At the same time, IL programme participants can be encouraged to tell their own stories to build up a nuanced and detailed picture of how people see the services on offer, where these services fit into people's own 'information worlds' and what real differences these services make, if any. These two types of story can be used in various ways:  Service case studies will show IL practitioners that their problems and successes are being recognised, give them a chance to signal other problems and provide a way to report their own good practice  User stories will help IL practitioners to understand better where they can make a difference and how  Service case studies and user stories should help managers of libraries to celebrate their successes, as well as to see what issues need to be addressed, where there are service development opportunities and what needs to be done next  Service case studies will give stakeholders (such as policy makers) concrete examples of how IL services are being provided, and user stories will show stakeholders what differences they can make to people's lives by supporting these servicesmaking the advocacy case  If you start collecting stories of both kinds early on in your evaluation work they can also help to form questions later in your evaluation. (Markless & Streatfield, 2013;p. 154, edited) Collecting and reporting individual and organisational stories of change is not new and, over the past ten years, edited stories of change have increasingly become an important element in advocacy on behalf of libraries, usually offered in combination with key performance data (see, for example, the Libraries Change Lives publication produced as part of the Public Libraries 2020 project -Reading and Writing Foundation, 2015).
Similarly, individual stories of change have been collected as an important part of impact evidence in many settings over the years. However, LIS leaders have shown a marked reluctance to gather and present stories of change except as examples to enliven evaluation reporting. There are probably two main reasons for this: the expectation that recipients of evaluation reports will dismiss stories of change as 'cherry-picked' to support the presenters' case and the amount of work required (and the associated cost) to assimilate many stories of change and present them in a 'balanced' report.
One interesting attempt to overcome this reluctance was undertaken in school libraries when the Ohio study led by Ross Todd and Carol Kuhlthau (2005), set out to answer the question 'How do school libraries/media resource centers help students with their learning in and away from school?'by asking them directly. The Rutgers University team in effect orchestrated a mass write-in among high school students in Ohio to an electronic questionnaire, enlisting the help of targeted schools and school librarians. More than 13,000 students responded, many of them giving powerful testimony to the value of school libraries, as reported by Whelan (2004). This project used the sheer number of responses to refute the challenge that people who responded to this write-in request were likely to be positively disposed towards their school library and therefore not necessarily representative of school students generally. http://dx.doi.org/10.11645/11.1.2201

What are the implications for IL practitioners?
If IL practitioners, LIS leaders and researchers wish to make better use of stories of change they may like to consider applying one of several possible approaches which have been offered to help overcome the 'cherry-picking' challenge and to enable evaluators to organise their stories of change. Here are three examples: Advances in computer software have provided an interesting approach to organisation of narratives (and other forms of evidence) is the SenseMaker  software offered by Cognitive Edge. In their words, 'The software and linked methods allow the collection and tagging of multiple sense-making items which can be anecdotes, pictures, web sites, blogs and other forms of unstructured data. These items can be also linked to more traditional systems such as content management. The tagging provides sophisticated metadata which can be used to provide quantitative research material, as well as measurement systems and impact analysis. Visualisation tools, linked to methods and models, permit users to sense complex patterns and anomalies that would not be visible to conventional analysis.' (This software has been used by various LIS organisations, such as INELI Oceania when evaluating their public libraries leadership development programme.) The Most Significant Change technique has gained many adherents since its introduction early this century by Dart and Davies (2003). MSC attempts to capture what matters in complex situations from the perspective of participants or, in this case, library service users. If applied in IL evaluation, organisers and participants would be asked to tell a positive or negative story about significant changes made to their lives by the IL work, and through a group filtering strategy, stories would then be chosen which best describe the essence of an intervention. This participatory technique is especially useful in situations where unexpected changes are likely, it builds organiser capacity to think about IL impact, and 'can deliver a rich picture' of organisational, social and economic developments (Davies and Dart, 2005). The authors claim that, because it is designed to provide information and stories of value to participants, it typically generates genuine interest and enthusiasm among those for whom the data are collected.
The third example is a general approach rather than a specific technique, and draws upon the democratic evaluation trend described in section 2 above. In this approach the LIS leader, IL manager, or evaluator actively seeks to engage representative organisers of and participants in the IL work when deciding selection criteria for stories, how much they should be edited and how the overall story should be presented (the formal report is not the only way). You may be able to set up virtual editing panels for this purpose. This approach provides a defence against cynical policy makers who may otherwise assume that you have only chosen stories that present the service in a good light. It also encourages participants of all kind to share in the key decision areas about evaluation, instead of leaving all the decisions (and the consequent power) to the person who organises the evaluation.

Looking forward
What changes to the world of information literacy evaluation are likely over the next ten years?
Here are a few predictions:  Picking up on the prevalent political climate in the Western World and on increasingly adroit manipulation of social media by politically motivated groups, it seems highly likely that there will be a strong focus on whether IL interventions are having an impact in combating misinformation and disinformation.
 We can expect to see more systematic and sustained approaches to impact evaluation in some sectors, for example in the health sector, where evidence-based practice is heavily emphasised, and in higher education, where there is growing emphasis on value for money. In these areas the use of coherent impact frameworks and ToCs are more likely to be implemented.  In services under pressure, such as many school libraries, little progress is predicted: library advocacy would be strengthened by local evidence of impact, but hard-pressed librarians will not be able to find time for this.  The more proactive public libraries are rethinking their service priorities to take advantage of technological developments and to combat the erosion of their traditional roles. This is likely to lead to greater engagement with IL, particularly for socially disadvantaged groups, as well as to a focus on the impact of such service innovations on users  Impact evaluation of IL provision in the workplace will depend on organisational priorities and on the importance placed on evidence by particular organisations.  Research into information seeking in context is likely to highlight areas of information literacy that should become the focus for LIS impact evaluation initiatives.
Finally, we hope that as current evaluation thinking becomes more widely understood by IL practitioners, they will move away from trying to claim that their work directly causes changes in users. Instead, they should be able to make a more convincing case for the contribution that IL work makes by employing programme theory-based evaluation.