Indicators of adult information literacy

Information literacy (IL), defined as the capacity to locate, evaluate and use information to create new knowledge, is a core adult life skill and an extension of the notion of functional literacy. The United Nations Educational, Scientific and Cultural Organization (UNESCO) has identified information literacy as an essential capacity for participation in the knowledge economy and has included this construct as a strategic priority in the Information for All Programme (IFAP). To support this initiative UNESCO has commissioned research to consider whether a household survey could be used to identify the extent to which people in member states are information literate in all aspects of their life, including work and lifelong learning, and in relation to their personal well-being, and in their participation in civil society. United Nations agencies use surveys to provide indications of the extent to which various agreed objectives are being achieved. These include international surveys of education and of health and well-being. This paper reports on an investigation aimed at determining whether indicators of IL could be identified by secondary analysis of existing data collected in other household surveys. It was concluded that IL has unique elements that are not encompassed in the surveys that were examined.


The need for information literacy indicators
Although information literacy (IL) is recognised as an essential skill for participation in the knowledge economy and in civil society (UNESCO 2003), IL has been a relatively neglected aspect of education policy.While library scientists have argued the merits of embedding information literacy in curricula (Bruce 2004), in many cases educators have been less than responsive.Therefore the development of indicators of IL at both the national and international level is an important initiative needed to focus curriculum managers and policy makers on the attainment of this competence.IL underpins many of the Millennium Development Goals (United Nations 2007).For instance, combating diseases (goal 6) and enhancing employment opportunities (target 16) require that people have the ability to apply information to practice.Indicators of IL can help countries to identify the effect of policies to foster or enhance IL development, and to know the extent to which their citizens are able to participate in a knowledge society.
Like other aspects of literacy, the level of IL skills required is dependent upon both the domain of practice and on the level of performance required of an individual.For instance, the level of IL needed for effective performance in elementary education differs from that required in adult and higher education.Different levels of skill are likewise required of a journalist and of a citizen in areas of civic involvement, and for health and well-being.http://dx.doi.org/10.11645/6.2.1746

The benefits of secondary analysis
The use of existing data sets to generate measures of related phenomena of interest has been widely recommended and is regularly supported by research councils (JISC 2011) and by research methodologists.Given the substantial costs involved in developing international indicator sets, Catts and Lau (2008) undertook a preliminary review of available international surveys of literacy components and concluded that there was a possibility that IL could be identified by secondary analysis of data collected in existing literacy surveys.They concluded that the primary source of such indicators could be the UNESCO Institute for Statistics (UIS)'s Literacy Assessment and Monitoring Programme (LAMP) survey.They recommended that UNESCO facilitate a content validity exercise of LAMP and other international or national household surveys to establish whether it were possible to select data elements that are valid indicators of IL.This approach would avoid the need and the cost of constructing a discrete IL survey.

Defining information literacy
Information literacy (IL) was described in the Alexandria Proclamation of 2005 as essential for individuals to achieve personal, social, occupational and educational goals (IFLA 2005).This statement argued that IL skills are necessary for people to be effective lifelong learners and to contribute in knowledge societies.Access to information is a human right and IL is essential in order for people to exercise this right.
The creation of new knowledge is an essential element in demonstrating IL and this capacity is required in order to become a knowledge worker.The expectation that most workers in all occupations can demonstrate IL is what distinguishes work in the 21 st century from at least some forms of work in previous generations.While there have always been knowledge workers this was thought of (rightly or wrongly) as the preserve of a small minority.These people stood out for their initiative and perception in identifying new ways of working or solving problems.However in the knowledge economy this capacity is a requirement for most workers in most occupations and hence IL has become an important competency for economic success.
For the purposes of the research initiative a definition proposed by Catts and Lau (2008) was adopted.This defines an information literate person as one that could:  Recognise their information needs  Locate and evaluate the quality of information  Store and retrieve information  Make effective and ethical use of information  Apply information to create and communicate knowledge.These five elements do not describe a linear process.Rather these are five inter-related elements.For example, a person may receive information and then recognise the potential utility for their situation, make use of the information and find it wanting, before evaluating the source and then recognising their need for reliable information.They may store information and subsequently retrieve it for use.Therefore while presenting five elements of IL, it is important to understand that they are inter-related elements of a single construct.This has been confirmed by factor analysis of responses to an Information Skills Survey (ISS) by university students (Catts 2005b).This study also established that among university students, five IL sub-scales could be distinguished by congeneric factor analysis (Catts 2005a).However, for government to monitor the IL skills of adults, IL indicators need to be developed in a wider context including for work, for lifelong learning, and for well-being.http://dx.doi.org/10.11645/6.2.1746 The development of information skills is important for individuals to operate effectively in a knowledge economy.Further, once a metacognitive level of IL is developed in one occupation, it is normally expected that the understanding of how to use information to create new knowledge can be applied in a different context, if the person is given suitable support to re-contextualise the knowledge and skills to the new setting.Thus IL can help to enhance the flexibility of the work force.In this sense IL is like other adult life skills and, as Singh (2003, p.4) argues, life skills are applied in all aspects of living and at all stages of living.

Comparing information literacy with other adult life skills
Adult competencies are normally considered to include literacy, numeracy, the capacity to work with others, communication skills, scientific literacy (or the use of technology), problem solving (sometimes called critical thinking), and information skills.Examples of each of these capacities can be delineated that are fairly distinct but nonetheless none of these capacities is independent of the others.For instance, working with others requires communication.Solving problems requires information and in many instances performance depends upon prose, document and numerical literacy.This means that IL is likely to be correlated with attainment of other adult competencies.It also means that some indicators of other adult competencies may provide evidence of IL.These twin issues raise challenges and opportunities for the development of indicators of IL.
Since adult life skills are an inter-related set of capacities, elements of IL might be evidenced by the secondary analysis of indicators of other adult life skills.The following section explores the relationship between IL and selected other adult life skills and examines published indicators of these capacities to identify the feasibility of secondary analysis to develop indicators of IL.

Differentiating literacy and information literacy
Adult literacy includes the requirement that people comprehend and use information conveyed using written materials.This may result in the creation of new knowledge, but knowledge creation is not an essential component to demonstrate literacy.For example, a literate worker can follow instructions and read safety signs (literacy) but need not know how to access new information in order to create new signs (IL).Literacy is a prerequisite for IL.
An analysis of literacy skills by occupation was undertaken by the Education Testing Service (ETS) and reported by Barton (1999).Based on the Position Analysis Questionnaire (PAQ) Barton provided an analysis of the changes in literacy requirements from 1986 with projections to 2006 and concluded that for 522 occupations (Barton 1999, p.5) there had been no significant changes in literacy skills for these occupations over the 20 year period of the digital revolution.While acknowledging that changes have occurred in how information is accessed, as data storage has moved from paper, to microfiche and then to digital records, Barton concluded that the required capacity to comprehend information had not changed for each occupation.While literacy requirements may not have changed, this analysis missed the substantive change in the nature of work in the past 20 years.Processes in most occupations have changed so that workers are now part of a culture of continuous improvement through knowledge creation.This transformation in work processes is the essential new IL competency required for participation in the knowledge economy (Levy 2010); (OECD 2000).It was concluded that standard literacy surveys were unlikely to be sufficient for secondary analysis to identify IL.

Differentiating ICT skills from information literacy
Information communication technology (ICT) skills encompass the capacity to use all forms of information communication technologies including computers, digital phones and other http://dx.doi.org/10.11645/6.2.1746 portable devices.The role of ICT has been the subject of surveys at national and international level including by the World Bank (World Bank 2011, p.v), which claims that ICT skills result in economic benefits such as "higher productivity, lower costs, new economic opportunities, job creation, innovation, and increased trade and exports".The World Bank report also suggested that ICT skills "help provide better services in health and education and to strengthen social cohesion".These claims seem to imply that at least some components of information literacy are included in ICT skills.Other reports explicitly widen the definition of ICT skills to include the impact of ICT, which is defined as involving the "use and production of content (in particular, electronic or digital content) which only exists because of ICT" (OECD 2008a, p.6).This is why the search for IL indicators via secondary analysis of other surveys needed to include ICT indicators because the capacity to access and to use ICT may demonstrate IL competencies.

Problem solving and information literacy
In the adult literacy and life skills survey reported by the Organisation for Economic and Cultural Development (OECD) and Statistics Canada (Murray et al 2005) five elements of problem solving were identified which, like IL, were considered to be inter-related elements that people utilise in a non-linear manner.These elements were described (OECD 2005b, p.303)This definition of problem solving appears to overlap considerably with some elements of IL and hence this construct was considered as a possible source of evidence of IL.Therefore sample problem solving questions were considered in the search for secondary indicators of IL.

Scientific literacy and information literacy
There does not appear to be an adult skills indicator of scientific skills or use of technology apart from the indicators of access to ICT.However, the OECD Programme for International Student Assessment (PISA) included the assessment of what was termed scientific literacy for school students.The OECD definition of scientific literacy overlaps with that of IL.In the PISA Report (OECD 2006, p.13) scientific literacy is defined in terms of four dimensions as follows: Scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, and to draw evidence-based conclusions about science-related issues.
An example of the overlap with IL is whether when individuals read about a health-related issue, they can they separate scientific from non-scientific aspects of the text, and distinguish reliable information on which to justify personal decisions.This involves the IL skill of evaluating information.Of course the PISA definitions are intended to relate to the life experiences of adolescents and hence PISA definitions and questions do not model adult life skills.For this reason PISA scientific literacy questions were not included in the content validation task.Media literacy has also been addressed by UNESCO initially by the development of media and information literacy (MIL) modules for initial teacher education (ITE).Consideration is now being given to MIL indicators which involve integrating media literacy and IL into a new construct that recognises the extent to which digital environments increasingly enable individuals to receive, transform, and transmit information.

Information literacy and media literacy
In the context of adult life skills, media literacy has been defined (Celot 2009, p.22) as: ...the capacity of individuals to interpret, analyse, process and contextualise media messages in general … media literacy implies a broadening … of the elemental function by which literacy is defined ….It is not a simple technical skill … but rather a critical understanding and analytic reading of numerous simultaneous sources of information, reasoning, social injunction, symbolic and cultural codes and conventions.
There is a significant overlap between this conception of 'media literacy' and IL.The author quoted above placed an emphasis on individual capacity to use media in the process of engagement in civil society and recommended that in the further development of indicators of media literacy there should be cooperation with UNESCO.Hence there is a substantial overlap between the concepts of media literacy and IL, but unfortunately no household surveys of media literacy were identified.Subsequently the European Commission has produced assessment criteria for media literacy (European Union 2009).Criteria are proposed at three levels and include elements for media users and organisations and hence this may be a source of ideas if UNESCO proceeds with MIL indicators.

The scope of the application of information literacy
The application of IL at work, for health and well-being, for lifelong learning, and for participation in civil society represents a substantial challenge for the development of robust indicators.This is why an investigation was commissioned to determine whether existing household surveys contained questions suitable for the assessment of IL.The alternative of creating universal indicators of IL skills at work, for health, for lifelong learning, and for civic participation would require new resources both at national and international levels.

Information literacy and work
The use of information at work is seen to be central to the notion of a knowledge economy.As Catts and Lau (2008, p.11) pointed out: IL is central to both the notion of a learning organisation and to the development of a competitive advantage for firms and for nations within the global knowledge economy.
While the economic value of an information literate workforce is understood, the corollary is that workers may question and strive to enhance not only the effectiveness of production processes, but also the ethics of practices at work including effects on the health of workers and consumers, the environmental impact of production processes, and the integrity of product claims (Field 2001).

Information literacy for health and well-being
There are two distinct areas where the use of information in health provision is of importance.The first is among health professionals where access to current research and best practice is of importance to the delivery of quality services.The development of the Cochrane Library (Cochrane Collaboration 2007), which is a global independent source of evidenced-based health information, is an example of this provision for health professionals.http://dx.doi.org/10.11645/6.2.1746 There is also an academic publishing tradition that aims to ensure the quality of information shared among health professionals and researchers.At the level of professional practice the use of online access to patient records in much of the developed world enables the development of informed team work.It is evident that the IL of health workers is addressed by the professions and therefore the information skill of health professionals was not included in the ambit of the survey of indicators.
The other crucial health issue is the right of individuals to access information about health and well-being and their capacity to evaluate information from various sources to determine its accuracy and relevance.This is a concern especially in poorer nations where literacy remains a challenge, and where access to information may be restricted especially in rural communities and among women.Grant (2002, p.1) has defined health IL in this context as "the capacity of an individual to obtain, interpret, and understand basic health information and services and the competence to use such information and services in ways which are health enhancing".This capacity includes the ability to distinguish between information from credible sources such as the World Health Organization (WHO) and information from individuals and firms with a vested interest in making claims for services and products.This capacity is applicable especially where individuals can use the internet to obtain information about their own health and well-being.

Information literacy and civil participation
The London School of Economics Centre for Civil Society defined 'civil society' on its web page (LSE 2009), now archived, as follows: Civil society refers to the arena of uncoerced collective action around shared interests, purposes and values.… Civil societies are often populated by organisations such as registered charities, non-governmental organisations, community groups, women's organisations, faith-based organisations, professional associations, trades unions, self-help groups, social movements, business associations, coalitions and advocacy groups.
Support for the notion of civil society by government is based in part on the assumption that participation by people in their community is central to both social and economic wellbeing.An informed public is thought to be a strong defence against the influence of extreme groups and the ability to evaluate sources of information is essential to protect against propaganda.A second reason for the interest in civil society is as a response to the new realities of international markets.In recent decades the world economy has become a dominant factor in domestic economies.For instance the power of the transnational economy has been demonstrated in the 2009 credit crunch where especially in Europe, large nations (Britain, France) and small (Iceland, Ireland), developed nations (Spain and Italy) and consequently developing nations, have all been caught up in an economic maelstrom.Informed participation in society is advocated as a political necessity to counter global economic forces in both developed and developing economies.This brings a certain synergy between information skills for work and for participation in society but is not necessarily going to result in outcomes that leaders might desire.

Information literacy and lifelong informal education
In the Alexandria Proclamation on Information Literacy there was a direct link made between lifelong learning and information literacy.The statement called for "all governments and intergovernmental organizations to pursue policies and programs to promote information literacy and lifelong learning" (IFLA 2005).Central to the notion of lifelong learning is the concept of an autonomous adult learner.As Usher et al (2002) pointed out, adult education traditions identify aspects of experience as legitimate elements of adult learning.However, for learning to be transformative, experience needs to be combined with new information http://dx.doi.org/10.11645/6.2.1746 acquired either from a facilitator or through the capacity of the individual to access, evaluate and transform information into new knowledge.This can be done by an information literate individual at work by accessing information from other workers, or in community by participation with others in civic activities.New information can also be accessed from web based sources.In all these cases adults need IL skills to be able to independently access, evaluate, store and apply new information.
Standardised IL surveys have been developed for use in formal education settings.For instance, in the United States a consortium led by library scientists, and including some faculty members and statisticians, developed a test of IL knowledge (O'Connor et al 2002) based on the Association of College and Research Libraries (ACRL) standards, and called 'Standardized Assessment of Information Literacy Skills' (SAILS).More recently the Education Testing Service (ETS) has also published a computer based survey that they term iSkills Test, formerly known as ICT skills (ETS 2008).In Australia an Information Skills Survey (ISS) has been published (Catts 2005b).As IL has been addressed in formal education contexts, the focus in this research was on IL in relation to informal and lifelong learning.

Methodology
The process adopted to identify indicators of IL from the secondary analysis of other surveys involved what is termed content validation.This is an essential step in the design of surveys or assessment tools and involves experts matching survey questions with specified competencies.In this case the specified IL competencies were those proposed by Catts and Lau (2008).These were used by UNESCO for the Information Literacy Indicators workshop.The survey questions were identified by following the approach advocated by Catts and Lau (2008).As outlined below, this involved a preliminary review by the author of a wide range of international surveys, together with more than 20 national household surveys used in Britain, which were typical of those used in many countries.

Levels of information literacy
It was important to consider not only whether survey questions located were evidence of IL, but also whether the collection of suitable questions covered a range of levels of IL practice.Indicators need to differentiate between the capacity to perform tasks at a basic or more advanced level because tasks may require varying levels of competence in different contexts.To undertake a content validation task of existing survey questions it was not necessary to definitively determine the level of IL signified by questions because this can be done as part of the statistical analysis of pilot studies to confirm construct validity.However it was important to establish that the complexity of survey questions is discernible, and cover a wide range of capacity levels.Hence, a definition of levels of IL capacity was developed for application in the content validation workshop.If the validation exercise indicated that there were no questions at any particular level, or alternatively identified that questions were ambiguous in terms of level of IL capacity, then this would suggest that secondary analysis to identify indicators of IL might not provide a sufficient range.
In the area of problem solving Kirsch et al (2005) postulated four levels of problem solving ability for the OECD Programme for the International Assessment of Adult Competencies (PIAAC) survey.These levels were adopted also in the Adult Literacy and Life Skills (ALLS) survey (OECD 2005a, p.306).As these OECD surveys were to be considered in the content validation workshop, these levels of difficulty were adapted to fit the definition of information literacy, and are as described below.
Level 1: At a very elementary level, concrete, limited application of information can be demonstrated by using specific schemata to assess the accuracy and relevance of information and to interpret the information in a specific context.http://dx.doi.org/10.11645/6.2.1746Level 2: The second level requires rudimentary systematic reasoning to access, evaluate, store and integrate information with prior knowledge.Information use at this level is characterised by well-defined, one dimensional applications.At this level, people apply information to concrete situations.Level 3: At the third level of IL proficiency people will be able to identify, evaluate, store and integrate multiple and potentially conflicting information sources and to apply information where there are non-transparent or multiple dependent constraints.Level 4: At the highest level of competency, people are capable of locating, evaluating, storing and applying information in situations that require sequences of actions and other "meta-features" in a systematic manner.Also, at this stage people are able to explain how and why they arrived at a certain conclusions.This level of IL requires critical thinking and meta-cognition.

Review of household surveys
Publicly available household surveys were identified from UNESCO, OECD and the Demographic and Health Survey (DHS).In addition international surveys used by other United Nations agencies were identified and considered but it was concluded that none had more than one or two questions that might also indicate IL.Survey questions were selected from four international household surveys that, in addition to their primary purpose of surveying literacy or ICT skills, appeared to indicate IL.British national household surveys were also considered as representative of data gathered in many advanced economies but again, no survey provided sufficient coverage of IL to warrant the time and cost of formal content validation of survey questions.The review of surveys conducted in Britain revealed that at best the data collected allowed broad inferences to be drawn about IL.For example, in Britain the Office for National Statistics (ONS) provides a report series called 'Social Trends' which includes the frequencies of fourteen uses of the internet (Randall 2010) but whether these involve ICT skills in receiving and transmitting information, as opposed to IL skills in transforming information, cannot be determined with the possible exception of 'doing an on-line course'.A few questions were presented from these sources during the induction to the workshop to help experts to differentiate IL from other adult competencies, but these questions were not subjected to formal procedures because they were manifestly not indicators of IL.
Household surveys can be classified into three groups; namely background surveys; selfreported behaviour; and pencil and paper (or computer-based) cognitive tests.Potential indicators were assembled from four surveys.The surveys selected for analysis were the UNESCO Literacy Assessment and Monitoring Programme (LAMP) background survey, elements from the OECD ALLS survey to which access was granted on a confidential basis, the Indonesian version of the DHS and the OECD ICT survey.These surveys were selected both because they provided examples of the various types of household survey identified above, and because they were considered to be the surveys most likely to yield questions suitable for secondary analysis of IL.Two of these surveys were reviewed by two pairs of experts, and the other two were reviewed by a composite team after one IL specialist had to leave for a prior commitment.
The nature of each of the four selected household surveys is described below.

LAMP background survey
The UIS Literacy Assessment Monitoring Programme (LAMP) survey is a development from earlier OECD surveys (UIS 2009) but is designed to be used in countries with vastly different infrastructure and levels of education.It consists of a background survey followed by questions that measure literacy, numeracy and problem solving and was under development.LAMP is a survey which was developed and is being used by UNESCO and it was therefore relevant to examine its potential as a source of indicators of IL.Only the LAMP background survey was included in this review to address copyright concerns.As this survey http://dx.doi.org/10.11645/6.2.1746 is similar in form to the background surveys in other international household surveys, it provided an opportunity to determine whether secondary analysis of questions in household background surveys could provide an adequate indication of IL.This was one of the surveys identified by Catts and Lau (2008) as being of potential utility for secondary analysis.

Adult literacy surveys
Over the past 15 years OECD has developed the assessment of adult literacies through a series of international surveys.The current version is called the Programme for International Assessment of Adult Competencies (PIAAC) which was implemented in 2011 across 27 countries with results to be reported by 2013.In PIAAC, ICT based problem solving indicators were added to the earlier versions of this survey."This refers to the ability to use technology to solve problems and accomplish complex tasks" (OECD 2008a, p.7).
There is an overlap between the definition of problem solving adopted by PIAAC and definitions of IL which made it important to consider the questions in the problem-solving domain as a possible source of indicators of IL.Questions from PIACC were unavailable so problem solving questions from an earlier version of the OECD adult literacy survey was used to investigate these questions as indicators of information validity.This leaves open the possibility that in the new version of these surveys there may be a resource that could be suitable for use to identify IL by secondary analysis.

Demographic and Health Survey
The Demographic and Health Survey (DHS) is a project supported by the WHO that has earned a worldwide reputation for collecting and disseminating nationally representative data on health and population in developing countries/regions.The project is implemented by Macro International, Inc. and is funded by the United States Agency for International Development (USAID).This makes the DHS a useful potential source of IL indicators because it could be used in conjunction with other agencies.The version selected was the Indonesian DHS Survey (Statistics Indonesia 2008).A feature of the DHS is that it is designed so that questions can be adapted to suit the conditions of use in each country, while the quality of the measurement is maintained by reliance on a standard administrative protocol.Some of the questions relate to the use of information for health related decisions.These questions were selected for consideration as a possible source of evidence of IL especially in relation to health and well-being.

OECD ICT skills survey
The final survey examined was the published OECD ICT skills survey.The capacity to access and to use ICT may include IL and for this reason selected questions from the OECD ICT access survey were considered.

Workshop protocol
The protocol adopted to demonstrate IL content validity was as recommended for test development by the American Psychological Association (APA 1999).The workshop was conducted at the UNESCO Institute of Statistics in Montreal in November 2009.The expert group included two information science specialists and two adult community education specialists and they were drawn from Canada, South Africa, New Zealand and the United Kingdom.It involved the experts identifying both the element and level of IL conveyed in selected survey questions.The question to be decided was whether there were sufficient valid questions to enable one or more existing surveys to be used for secondary analysis to indicate IL.
The four experts had taken part in an initial training activity using some questions from a UK national survey and from the DHS survey, to ensure all were familiar with both the standards and the content validation process.They then worked in pairs to review each selected survey question.Initially each expert made their own rating of each question to determine whether IL would be indicated by each question, and if so at what level of difficulty.It was http://dx.doi.org/10.11645/6.2.1746 agreed that IL could be identified as an explicit outcome of a question or as a necessary but implicit aspect of the question.Once each expert had made an individual assessment of a group of survey questions, they compared their judgements with their partner.Where they agreed their decision was confirmed.Where there was some measure of agreement, the differences were discussed, and a decision was reached with comments added for later review.
The following survey question illustrates the process.The question stated: 'During the past month, how often have you used a computer to write or edit texts?'The answer 'Never' may suggest that the person cannot do this task, but it could equally suggest that the person has had no access to a computer.Furthermore, editing may require skills in formatting text without the need for evaluating the content, or it may require a detailed grasp of the content and the capacity to critically appraise and modify the text.Hence the level of complexity of the implied IL cannot be identified.These twin reservations meant that what was a promising question on first sight was not content valid for secondary analysis.
After rating an initial set of survey questions the outcomes were reviewed by the expert group as a whole.Differences in the ratings between experts were discussed and resolved.This process consolidated a shared understanding of the interpretation of the elements and levels of IL.There were problems evident in the use of the survey questions for secondary analysis of IL.To avoid initial negative outcomes impacting on expectations about the success of the analysis overall, the participants were reminded that there were three different types of surveys to be considered and as a consequence the failure of one type to provide valid indicators would not necessarily mean that success would not be achieved with another form of question.

Research outcomes
A total of 104 distinct questions were examined.The experts considered that IL was not evidenced in 43 of these questions.For the remaining 61 questions, the experts agreed that 29 questions conveyed explicit indicators of IL and that 23 questions conveyed an implicit indication of IL.For the other nine questions the experts agreed that IL was indicated but could not agree whether the construct was implicit or explicitly identified.However when the 61 questions were classified in terms of element and level as indicated in Figure 1, only 29 were identified as content valid questions of IL.In other cases there was no agreement on the element or the level of IL. element of IL at the preliminary level for IL.All of these 18 questions were drawn from the OECD ICT access survey.All the questions were indicators that people could locate information in concrete situations at a basic level using an explicit protocol.The ICT survey also yielded three questions that addressed the effective and ethical use of IL at the second level, namely rudimentary systematic reasoning to access, evaluate, store and integrate information with prior knowledge in concrete situations.Hence there were questions that addressed two of the five elements of IL, at two levels of complexity.While the ICT access survey demonstrated the best potential for identifying IL, it did not offer sufficient coverage of the construct to generate a valid set of indicators of the whole of the IL construct.

Figure 1: Level and element of information literacy in indicator items
The OECD ALLS problem solving questions were reviewed by three members of the expert team.Five of the ALLS questions provided an explicit indication of an identifiable element of IL.Three questions were confirmed as indicators of creating and communicating information at an advanced level (3 or 4).One involved evaluating information at an intermediate level (2), and one involved using information at a basic level.The panel concluded that ALLS problem solving questions could not provide a sufficient set of indicators of IL.This finding is understandable since ALLS is focused on literacy and problem solving and not on IL.The elements of IL not addressed by ALLS were the abilities of respondents to identify information needs by searching for information, evaluating the information, and the storage and retrieval of information.
The background survey used for LAMP is a high quality example of the types of questions posed as background questions in many household surveys.Consequently the analysis of these questions provided an indication of the suitability of background questions in other household surveys.Forty-six elements of the background survey were considered by two teams of reviewers.It is noted that some of these questions contain multiple components so the total number of discrete questions is greater.Of the 46 questions, seven were identified as possible indicators of IL but in four cases the question was considered not suitable because the level of the performance in this self-report format could not be assumed.Consequently just three questions in the background survey were considered to be suitable indicators of IL and this was an insufficient number to use to establish a reliable scale for secondary analysis.One question was agreed to represent IL awareness at level 1; another involved evaluation of information at a basic or intermediate level, and a third addressed the use of information at a more advanced level.
No questions from the DHS survey (used as the training survey) were considered suitable because where IL was identified it was considered in most cases to be implied rather than overtly addressed and hence neither the elements nor the level of IL could be agreed.

Discussion
The questions subjected to this analysis are all questions that were developed for related adult competencies including literacy, health and well-being, problem solving and ICT skills.The research was not a review of the content validity of questions for the purpose for which they were designed.Consequently finding any question to not be content valid for IL is not a criticism of the content validity of questions in relation to the purpose for which they were designed, and findings do not invalidate the use of questions for their primary purpose.It means only that the questions are not valid for secondary analysis as indicators of IL.
Consideration of the problem solving questions from the ALLS survey demonstrated that for the identification of IL skills a pencil and paper test has limitations because it does not require respondents to seek and evaluate new information.The ALLS survey required people to make inferences beyond the information provided, but IL requires the person to go further and to recognise the need to locate and evaluate additional information.Therefore important elements of IL, including awareness of the need for information, the capacity to locate information, to evaluate the quality of the information and its sources, and the capacity http://dx.doi.org/10.11645/6.2.1746 to store and retrieve information, were not included.A pencil and paper format can be appropriate for problem solving and may indicate the capacity of a person to use and interpret information which is one element of IL, but without requiring evidence that the person can access, evaluate and select additional information this represents a limited use of information that does not incorporate the full meaning of IL.
The opportunity to identify IL from the secondary analysis of established international surveys was worth the effort, as success would have avoided the need to develop a new survey.The conclusion that there are limitations in the validity of questions from other surveys for the purpose of identifying indicators of IL is not a criticism of these surveys, but a demonstration that an adequate set of IL indicators cannot be identified by secondary analysis.Although the PIAAC assessment results to be released in 2013 may offer more synergy with indicators of IL, the focus on cognitive skills means that core aspects of IL including values and attitudes may be excluded.
In different regions of most countries economic and social conditions and access to infrastructure vary considerably.Indicators must therefore be robust for use across a wide diversity of conditions.These challenges have been addressed to varying degrees in the design of each of the household surveys considered.There may however be an additional element to consider in an IL household survey, namely how to identify information search strategies, as these may differ depending upon the level and form of access to media and to telephones, including mobile phones.
as follows:  Search for information, structuring and integrating it into a mental representation of the problem  Reasoning based on the situational model  Planning actions and other solution steps  Executing and evaluating solution steps  Continuous processing of external information and feedback.