Instructional elements in an online information literacy Open Educational Resource (OER) and their influence on learner achievement, satisfaction and self-efficacy

This study tested the influence of instructional elements within an online Open Educational Resource (OER) focused on information literacy (IL) on outcome measures of IL achievement, learner satisfaction and IL self-efficacy among undergraduate students. An online OER was designed to address the domains of access, evaluation and communication of IL guided by the notion of instructional scaffolding and self-regulated learning. Participants were randomly placed into one of six different OER conditions: (a) full version with all instructional elements, (b) lean version, (c) version without tooltip text, (d) version without embedded practice questions, (e) version without learning objectives and (f) version without summaries. There were no significant differences found across the six conditions on the dependent measures. Participants averaged 58% for IL achievement, performing slightly better in the domain of access versus evaluate and communicate. Limitations include a controlled laboratory setting where participants were not necessarily motivated to complete the study tasks at a high level of achievement. Future research can explore more ecologically valid environments where learners might be more motivated, along with more rigorous intervention and assessment construction. This paper includes implications for educators and researchers to explore the established and innovative instructional elements that are natural affordances of an online OER in IL. This paper presents innovative IL instruction that does not require instructor or learner training and evaluates its effectiveness using a sound, replicable methodological approach to isolate the effects of the individual instructional elements.


Instructional elements in an online information literacy Open Educational Resource (OER) and their influence on learner achievement, satisfaction and self-efficacy 1. Introduction
Educators in higher education have an essential task in designing instructional activities that engage learners and effectively help them gain information literacy (IL) knowledge and skills (Association of College and Research Libraries [ACRL], 2015). This is especially important in the contemporary online environment as informationor misinformationcan be purposefully or carelessly spread, leading to serious and dangerous social and political consequences (Bastick, 2021;Levitin, 2017;Wineburg & McGrew, 2018). Indifference to facts has also been popularised in this era (Reed et al., 2019), making IL skills more essential now than ever. As online learning in higher education has been considered 'potentially transformative' (Goodman et al., 2019, p. 1), the researchers chose to create an Open Educational Resource (OER) designed to be used as a stand-alone IL learning experience that can be delivered in the online environment. This study explored how various instructional elements of IL delivered through an OER (tooltip text, embedded practice questions, objectives and summaries) influence IL achievement, learner satisfaction and IL self-efficacy.

Review of relevant literature
IL presents a complex and overwhelming problem for researchers and educators due to a combination of multiple factors. These factors include the overload of users and information online, the lack of direction in terms of addressing the problems through instruction, and the serious potential consequences of information (or misinformation) being carelessly spread. In this study, the researchers attempt to address this complex issue through the affordances of an online IL OER delivered to higher education learners. The following sections will discuss some of the literature on key topics pertinent to this study.

IL in Higher Education
IL literature establishes the value of IL education as part of the university academic experience for learners (Schmidt Hanbidge et al., 2018). Learners need to know 'how to conduct research and be self-reliant in the electronic information environment' (Schmidt Hanbidge et al., 2018, p. 118). In higher education, many institutions are prioritising the implementation of high-quality IL instruction embedded within, or as a supplement to, existing curriculum (Anderson & Mitchell, 2012;Hsieh et al., 2014;Mullins, 2014). Often, academic libraries have been called upon to provide IL instruction (Nichols Hess & Greer, 2016). This has been seen, with varying results, to come in the form of academic partnerships with faculty, one-shot sessions and for-credit IL courses (Nichols Hess & Greer, 2016).
The content of this instruction varies, and library associations have emphasised adaptable approaches to IL rather than a standard curriculum. For example, in Europe, the Society of College, National and University Libraries (SCONUL) created a Core Model (Bent & Stubbings, 2011) of IL with seven 'pillars' that represent essential knowledge and skills of IL, along with multiple 'lens' that can be applied to different user communities. The goal is that 'the model can be used flexibly by individuals and teachers who can adapt it as appropriate to personal circumstances' (Bent & Stubbings, 2011, p. 4). Similarly, in the U.S. the ACRL overhauled their previous definition of IL in the Framework for Information Literacy for Higher Education (2015). This updated document focuses on threshold concepts and learners as producers of information, replacing the skills-focused standards of the past. Like SCONUL's Core Model, ACRL's (2015) Framework was presented as a tool to encourage conversation, with librarians and faculty at individual institutions responsible for determining the best plans for implementation (2015).
Despite these expanded definitions of IL, the core IL skills of accessing, evaluating and communicating information remain foundational to IL. This is demonstrated in ACRL's (2015) description of IL as 'the set of integrated abilities encompassing the reflective discovery of information, the understanding of how information is produced and valued, and the use of information in creating new knowledge and participating ethically in communities of learning.' (ACRL, 2015, p. 3).
Yet, with academic librarians being called to engage in flexible, context-dependent approaches to IL instruction, the foundational skills of information access, evaluation and communication may be better taught in an online format through OERs.

Online learning and OERs
OERs are 'educational materials either licensed under an open copyright license or in the public domain' (Wiley et al., 2014, p. 781). While not exclusively digital, many OER implementations have naturally been deployed in the online environment, including many open textbooks and interactive online learning resources. Online OERs can be shared, distributed and repurposed, depending on the nature of the open copyright license, to learners and educators across institutions and international borders resulting in major cost savings for students and more quality options for educators to draw from in their classrooms (Al Jamil et al., 2019;Mery et al., 2012).
As technologies advance, online OERs have also become increasingly effective, providing instructional opportunities that only used to be attainable through face-to-face instruction (Bonk & Graham, 2006). OERs in the online learning environment can provide learning opportunities that are independent and self-directed, while encouraging lifelong learning (Kiliç-Çakmak, 2010). Online OERs can improve instruction, individualize experiences and decrease the costs of learning (Harkins et al., 2011). As a result, OERs have grown in popularity (Al Jamil et al., 2019;Lee & Ferwerda, 2017).
Similarities can be drawn between the OER developed in this study and online library tutorials that focus on IL. Like OERs, online library tutorials can provide learners with instruction efficiently while also providing a tool that libraries can re-use to deliver instruction to more individuals, regardless of location (Anderson & Mitchell, 2012;Beile & Boote, 2004;Greer et al., 2016). These tutorials help address issues with limited resources and have been shown to be effective delivery methods for library instruction (Beile & Boote, 2004;Greer et al., 2016). For the purposes of this study, we considered our instructional resource to be an OER instead of an online library tutorial because much of the content was not directly related to library tools, and the instruction was not tied to any library in particular. Also, the learning materials are not housed on a library website, but instead posted on one of the researchers' personal websites for open use and dissemination. Our resource can be most closely categorised as a short ebook, and the term OER is overarching and includes open textbooks (Wiley et al., 2014). Therefore, we consider our instructional resource to be an OER designed to foster IL skills in higher education.

OERs and online instructional scaffolding
The digital technologies used to create and deploy online OERs offer both instructional (like learning objectives) and technological (for example embedded practice) affordances to students. In the online learning environment, an educator or expert may not be available to provide additional guidance to learners, but instructional and technological affordances can provide instructional scaffolding and support self-regulated learning (Azevedo et al., 2004;Sharma & Hannafin, 2007) despite the instructor's absence.
Instructional scaffolding was initially described as assistance from experts that enables students to achieve what is beyond their ability to accomplish alone without support (Wood et al., 1976). Grounded in Vygotsky's notion of the Zone of Proximal Development (ZPD) (Vygotsky, 1978), scaffolding can offer students support that can be gradually faded until the students are capable of performing the tasks independently. With the advent of digital technologies, instructional scaffolding has been operationalised in technology-enhanced learning environments, including online OERs. OERs in the online environment can provide a range of instructional elements to support student learning experiences, such as taking traditionally passive learning activities (like reading a textbook) and turn them into more interactive experiences that allow students to reap the benefits associated with active learning (Fenwick et al., 2013). However, since these resources may not be facilitated with the guidance of an expert educator (especially in the online environment), students will need to engage in self-regulated learning to reap their benefits. OERs can use instructional and technological affordances to encourage forethought, performance and reflection (Zimmerman, 2002). For example, these affordances can encourage learners to conduct strategies like goal setting, self-instruction and self-evaluation.
Instructional scaffolding theory classifies technology-enhanced scaffolds in a variety of ways in the research literature. To inform our implementation of the instructional elements used in this study, we adopted three of these classifications. First, we consider the distinction between hard and soft scaffolding, where hard scaffolds refer to pre-planned static forms of support that are designed to help students through anticipated difficulties, and soft scaffolds are typically provided on an as-needed basis and are customised, dynamic and negotiable (Sharma & Hannafin, 2007;Shin et al., 2020). Second, we note the distinction between embedded and non-embedded scaffolds in which an embedded scaffold is placed within the learning environment (like embedded practice), and non-embedded scaffolds (such as peer support) are initiated by the students themselves (Saye & Brush, 2001). Finally, Hill and Hannafin (2001) classified technology-enhanced scaffolds for open-ended technology-enhanced learning environments as conceptual, metacognitive, procedural and strategic. We provide examples and definitions of each in Table 1.

Conceptual
Assists the student in deciding what to prioritise and what is important for their learning. Learning objectives can provide this support to students by providing a pre-instructional activity to focus their attention and provide guidance for preparing to learn new materials.

Metacognitive
Helps learners assess what they know and what they need to learn. Embedded practice or summaries can provide this support for students by allowing the students to practice on relevant tasks, and monitor and reflect on their own performance at appropriate points in the learning experience.

Procedural
Help learners use the resources available to them in the learning environment.
Providing detailed instructions and supports on how to use specific tools (such as navigation or software utility) and resources available to students in the learning environment to enhance student learning outcomes.

Strategic
Provide students just-in-time support or guidance for difficulties they are facing in a learning situation at a given instance in time. The use of tool tip text provides definitions of complex vocabulary words in educational text contents in hypermedia on demand, which could serve as a hints or advanced organisers to support learning outcomes.

Affordances of OERs in IL instruction
Online OERs have frequently been used to deliver IL instruction that reaches learners in innovative ways (Nichols Hess & Greer, 2016). Because the internet is central to IL, online OERs offer an ecologically valid paradigm of teaching and learning (Allen, 2008). OERs that build in scaffolding and self-regulated learning are also ideal for IL since the ability to evaluate information is primarily a self-monitoring activity aligned to the goals set by the student (Greene et al., 2010). Furthermore, teaching foundational IL skills through an OER eliminates barriers for students who may not be able to easily access the library (Harkins et al., 2011). Schmidt Hanbidge et al. (2018 argued 'it is especially important that students learn how to conduct research and be self-reliant in the electronic information environment at a time when there is less need to consult directly with a librarian or to walk into a library ' (p. 118). OERs may support this student independence.

Instructional elements in OER on IL
This study focused on the influence of four specific instructional elements as instructional scaffolds, delivered via an online OER, and their effects on IL achievement, learner satisfaction and IL self-efficacy. The instructional elements of interest in this study were chosen based on previous literature drawing on inspiration from instructional scaffolding and self-regulated learning in online learning, hypermedia and MOOCs (Martin & Klein, 2008;Zheng, 2016), and the affordances of the WordPress platform in which the OER was deployed. The instructional elements of interest were tooltip text (leads), embedded practice questions, learning objectives and summaries. These instructional elements will be briefly described below. All of our scaffolds can be characterized as 'hard' and 'embedded' since the students can engage with these scaffolds directly in the online OER, and since these instructional elements are pre-planned to address specific outcomes and not customised, dynamic and negotiable based on adaptive sources from the students.

Tooltip text
Tooltip text, or what is often referred to as leads (Antonenko & Niederhauser, 2010), is a portion of text in which extra information appears as a pop-up when the learner's mouse hovers over it (Hansen et al., 2009). In this OER, tooltip text was used to provide definitions of keywords throughout the text; definitions would appear when students hovered their mouse over underlined keywords. This is common in online OERs, as tooltip text functions used for vocabulary are included in many configurations where dictionary definitions pop-up if users hover over words (Lee & Lee, 2015). Providing definitions within the text can assist in reading comprehension and vocabulary learning, while also helping readers when words or text complexity are beyond their knowledge or abilities (Varol & Erçetin, 2019). This type of scaffolding could be classified as strategic because it provides just-in-time support and serves as an advanced organiser for the students.

Embedded practice questions
Practice multiple-choice questions that did not contribute to any scores or grades were included throughout the OER. These provide learners the opportunity to perform (Gagné, 1985), then receive feedback that can confirm correct answers or signify incorrect understanding (Martin & Klein, 2008). These practice questions were included at relevant points in the OER, following sections in which learners were provided information to master key objectives from the textual content (Martin & Klein, 2008). This can provide many benefits for learners, including strengthening new knowledge that learners are obtaining (Foshay et al., 2003) and improving retention (Kruse & Kevin, 1999). It has also been found that practice is effective when used in the online instructional environment (Martin & Klein, 2008). This instructional element can be characterised as a metacognitive scaffold since the students are able to engage in distributed practice with formative feedback to self-evaluate their mastery of the IL topics and self-regulate in response (such as re-reading a section after answering a practice question incorrectly).

Learning objectives
Learning objectives were presented at the top of each content page of the OER. These objectives described the specific outcomes that learners are intended to attain following the contents on that page in the OER (Mager, 1962). Providing learning objectives at the beginning of instruction helps learners structure learning (Ausubel, 1968), self-regulate and guide their learning of the IL content (Reiser & Dick, 1996) and enhance relevant learning overall (Martin et al., 2007). Learning objectives can assist students in activating prior knowledge, planning for the learning experience and monitoring the goal-attaining process (Azevedo, 2005). Also, learning objectives have been found to improve learning in online environments (Klein & Cavalier, 1999). As learning objectives can guide the students on goal-setting and serve as a pre-instructional strategy to help them anticipate the new materials to be learned, this type of instructional element can be classified as a conceptual scaffold in the context of an online OER.

Summaries
Summaries, or 'an outline of the key information that was presented to learners' (Martin et al., 2007, p. 632), were included at the end of each content page in the OER. These summaries were presented to learners in bullet-point format. Summaries, also referred to as reviews, can reinforce what the learners were supposed to attain in the instruction (Reiser & Dick, 1996), reassuring learners that they are understanding the content that was just presented (Mattiske, 2001). Using summaries has shown evidence to enhance learning (Hartley & Davis, 1976) from expository texts. Van der Zee et al. (2018) found that having learners read summaries in the online environment were related to increased performance. Even meta-analyses show that review as a process has significant positive relationships with metacognitive and cognitive learning strategies and academic performance across grade levels and academic disciplines (Dent & Koenka, 2016). Reviews or summaries can serve as a self-evaluation mechanism to assist students in activating the self-reflection process of self-regulated learning, characterising it as a metacognitive instructional scaffold.

Control instructional elements
While our present research design exclusively focuses on the four instructional elements, our online OER on IL manifested other relevant scaffolds found to be effective in improving student learning outcomes. For instance, we provided explicit guidance on how to use each of the instruction elements at the start of the learning process, which is a form of procedural scaffolding (Sharma & Hannafin, 2007). Additionally, we provided carefully and meaningfully chunked sections of expository text (Gobet, 2005) incorporating signaling principles such as appropriately labeled headers (Schneider et al., 2018) and logically sequenced expository text (Hebert et al., 2016) using a range of text cues (including bold or bullets) with useful examples (Mayer, 2020) and tables (Roehling et al., 2017). We also employed external hyperlinks to relevant website tools on IL (fact-checking websites like www.snopes.com) (Al Mamun et al., 2020), and an active hyperlinked table of contents and open-ended navigational system serving as learner control (Karich et al., 2014). These additional instructional elements were available across the content domains of access, evaluation and communication in the OER, and thus were treated as constants in the research design. Figure 1 provides a sample screen shot of the OER to illustrate these instructional elements available to the students.

Purpose and research questions
The purpose of this study is to explore how various instructional elements in an online IL OER (tooltip text, embedded practice questions, learning objectives and summaries) influence IL achievement, learner satisfaction and IL self-efficacy in the online environment. Our guiding research question is as follows: How do tooltip text, embedded practice questions, learning objectives and summaries in an OER on IL influence achievement, learner satisfaction and IL self-efficacy for undergraduate learners in the online environment?

Research design
This study explored how various instructional elements in an OER (tooltip text, embedded practice questions, learning objectives and summaries) influence IL performance, learner satisfaction and IL self-efficacy in the online environment. In order to do this, this study employed an experimental design consisting of six conditions: • Full interventioninstruction consisting of all instructional elements noted.
• Lean interventioninstruction consisting of none of the instructional elements.
• No tooltip textincluded all elements except tooltip text.
• No embedded practice questionsincluded all elements except practice questions.
• No learning objectivesincluded all elements except learning objectives.
• No review summariesincluded all elements except review summaries.
The goal was to isolate each one of these instructional elements to see if their presence (or lack thereof) influenced learner achievement, satisfaction and self-efficacy. The OER was built on a WordPress platform and the final intervention used in this study was offered as a Qualtrics online survey. Participants were able to simply navigate to the survey on a desktop computer, where Qualtrics would randomly place the participant into one of the six conditions.

Participants
Data was drawn from participants in various undergraduate courses offered by the College of Education at a large southeastern research university. Courses were selected for the diversity of enrolled students in terms of age, major and year classification. Only data from participants who completed the entire Qualtrics survey (demographic survey, IL assessment, satisfaction survey, self-efficacy survey), a total of 253 participants, were retained.

Intervention
The intervention used in this study was an online OER designed based on the American Library Association's (ALA, 1989) definition of IL that was included in the Information Literacy Competency Standards for Higher Education: a set of skills that allows individuals to 'recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information' (ACRL, 2000, p. 2). As stated earlier, the ACRL definition later changed to become broader and more applicable to the dynamic nature of the online environment; however, the three skills included in the previous definition (access, evaluate, use) remained foundational to the updated definition. While the ideas included in the updated definition played an integral role in informing this study, the previous definition provided the researchers with a more suitable approach for designing the resources. Based on this, the research team decided to build the intervention around three essential skills: accessing information, evaluating information and communicating information.
Using a backwards design approach (Wiggins & McTighe, 1998), the researchers first developed the learning objectives and the assessments. Three different sets of objectives and multiple-choice assessments were created, one for each essential IL skill drawn from the ALA (1989) definition. These objectives and assessments were built based on various information found through a systematic process exploration and curation by the research team, and a quality check was conducted by one team-member who is a subject matter expert in the area of IL. Based on the objectives and assessments that were developed, the research team wrote expository texts for each domain (access, evaluation and communication). These texts would act as the main source of instruction for the three essential skills and make up the content of the reading-based OER. A quality check of the narratives was conducted within the team (team members checked texts written by other team members), including the subject matter expert. Appropriate edits were discussed and applied until the team agreed about the three domains, establishing the OER on IL. It was decided to keep the content of the OER as simply reading-based instruction to establish a baseline in which to then compare and evaluate the influence of the instructional elements of interest on the outcome variables.
Once the objectives, expository texts and assessments were developed, the research team created and added the instructional elements of interest. These instructional elements were included via various OER instructional elements, which can also be thought of as instructional scaffolds (Pierard et al., 2019). These included tooltip text (keyword glossary items highlighted with a definition displayed when a learner hovered over the word with their mouse), embedded practice questions (relevant questions placed in deliberate locations in the instruction to let the learner test their understanding), learning objectives (these were already developed to begin each of the three domains of instruction) and review summaries (a bulleted list of essential takeaways presented at the end of each page of instruction). A WordPress instance was created to house the instruction, with various plugins used for the instructional elements. The four instructional elements integrated into the e-book are visualised in Figure 2: tooltip text, embedded practice questions, learning objectives and review summaries. Six different forms of the website were carefully developed for the six different conditions in the study.

Figure 2: Visualizations of the instructional elements integrated into the OER
The final product of the intervention was accessed as an external link in the Qualtrics survey used for data collection. Participants would first respond to the demographic survey and after be randomly assigned to one of the six WordPress instances. From there, they would navigate through an introduction home page with instructions on how to use each instructional element, an Access page, an Evaluate page and a Communicate page. The instructional reading usually took around 30 minutes to complete. After, the final webpage asked participants to close the WordPress site and continue in the Qualtrics survey, which would administer the Information Literacy Assessment (ILA), Learner Satisfaction Survey (LSS), and the Information Literacy Self-Efficacy Scale (ILSES). The full intervention with all instructional elements included amounted to more than 8,000 words of text with a Flesch-Kincaid Grade Level of 12.6.

Demographic survey
This survey collected sex, major, year classification, ethnicity and age. It was used to learn more about our sample and the population it represents.

Information literacy assessment (access, evaluate, communicate)
This was a multiple-choice test with N = 36 items (Access n = 12, Evaluate n = 15, Communicate n = 9) designed to assess various levels of IL learning (Anderson & Krathwohl, 2001;Bloom & Krathwohl, 1956). Based on the updated Bloom's taxonomy (Anderson & Krathwohl, 2001), items ranged from the Remembering level ('Paraphrasing is the act of _______.') to the Evaluating level ('Look at the screenshot below. How would you evaluate the information from this source?'). The assessment was designed by the research team based on the three constructs extracted from the ALA's (1989) definition of IL (ACRL, 2000).
Team members conducted their own exploration of IL principles in order to develop the questions in alignment with the narrative on each topic. After multiple rounds of quality checks among the research team, the assessment was piloted on a group of 14 undergraduate learners from the same population as the study's sample. Based on item difficulty and item discrimination results (Crocker & Algina, 1986) from the piloted assessment, the research team refined items to ensure their validity. Internal consistency reliability for the sample's scores 36item final version of the ILA was measured as KR-20 = .792.

Learner satisfaction survey
The LSS measured participants' affective feelings about the instruction. Participants were instructed to 'Please select the position on the scales below that best describes your impression of the instruction', followed by nine items that presented two opposite viewpoints about the instruction with a five-point Likert-type scale (such as 'Annoying-Pleasing', 'Unsupportive-Supportive'). This survey was validated in a previous study (Ritzhaupt, 2019), and Cronbach's alpha for the sample's scores was measured as α = .923.

Information literacy self-efficacy scale
This survey measured self-efficacy for IL. It is a 28-item scale developed and validated by Kurbanoglu, Akkoyunlu and Umay (2006) with the initial stem 'I feel confident and competent to', followed by various statements pertaining to abilities regarding IL such as 'Synthesize newly gathered information with previous information'. Participants responded on a seven-point Likerttype scale. Cronbach's alpha for the sample's scores on this survey was measured as α = .973.

Procedures
Once the intervention was fully designed and developed and the instruments were prepared in the Qualtrics survey, a team member contacted various instructors of different courses in the College of Education to see if data collection during course meeting times would be possible. Four different courses agreed to permit data collection in the Fall 2019 and Spring 2020 semesters, with a total of sixteen sections (individual groups of students within these courses) participating in the study. A script was developed so that individual differences among members of the research team would not be a factor during data collection. Two members of the research team taught courses in which data was collected; for these situations, a different member of the research team would collect data from those classes to avoid a conflict of interest. Instructors had the option of providing extra credit in their course to their students for participating in the study.
The script had the research team members express the details and purpose of the study in a straightforward manner, mention that participating is voluntary and optional, and give the participants the web address to access the informed consent form, which was Qualtrics survey. The Qualtrics survey research was designed for participants to be able to navigate in the online environment without any other instruction. Participants were instructed through the Qualtrics survey to complete the demographic survey, open the external link to take part in the intervention (reading the instruction), close the link to the readings, complete the ILA, complete the LSS and complete the ILSES. Team members administering data collection simply kept track of time for the participants and were available for any questions that participants had. Team members collecting data also had the task of making sure participants closed the reading (presented as an external link in the Qualtrics survey that took participants to the WordPress website) prior to completing the ILA in the Qualtrics survey. Data collection typically took around 50 minutes for participants to complete.

Data analysis
Descriptive statistics were computed for the outcome variables (performance on Access, Evaluate and Communicate) constructs of the ILA, performance on the ILA as a whole, responses on the LSS and responses on the ILSES within each condition. Along with this, a one-way Analysis of Variance (ANOVA) was run to test for group differences among the conditions based on the scores of the outcome variables (Keith, 2014). The assumptions of ANOVA were assessed prior to use of the statistical method, including normality, homogeneity of the variance and independence of observation. The researchers also used Levine's Test for Equality of Variances to test if homogeneity of variance was equal across groups (Brown & Forsythe, 1974). Finally, Pearson correlation coefficients were computed across the dependent measures of interest to illustrate the relationships among the measures.

Results
The descriptive statistics for the ILA achievement, LSS, and ILSES are shown in Tables 3 -5. As can be quickly gleaned, the mean scores across the six conditions did not vary widely for any of the observed measures. There were no severe departures from normality for any of these measures, and the independence of observation was achieved using the random assignment procedures carefully described in the procedures section. The homogeneity of variance assumption was checked with the Levene's test for each measure, and none suggested this assumption was violated. Thus, the data appeared to be well-suited for ANOVA.

IL achievement
The assumption of homogeneity was met for the ILA, indicated by Levene's Test of Equality of Error Variances at F(5,247) = 1.090, p = .367. Differences in scores on the ILA between conditions was not significant at F(5,247) = 1.154, p = .332. There were no significant differences in scores between conditions for any of the constructs within the assessment (Access, Evaluate, Communicate) at F(5,247) = 1.633, p = .152, F(5,247) = .801, p = .550, F(5,247) = 1.026, p = .403, respectively. For one-way ANOVA results and descriptive statistics for the full assessment and its three constructs, refer to Table 3. As the results were not statistically significant, the use of a post-hoc procedure was not employed.

Learner satisfaction
The assumption of homogeneity was met for the LSS, indicated by Levene's Test of Equality of Error Variances at F(5,247) = 1.075, p = .375. Differences in scores on the LSS between conditions was not significant at F(5,247) = .319, p = .901. For one-way ANOVA results and descriptive statistics for the LSS, refer to Table 4. As the results were not statistically significant, the use of a post-hoc procedure was not employed.

IL self-efficacy
The assumption of homogeneity was met for the ILSES indicated by Levene's Test of Equality of Error Variances at F(5,247) = .528, p = .755. Differences in scores on the ILSES between conditions was not significant at F(5,247) = .780, p = .565. For one-way ANOVA results and descriptive statistics for the ILSES, refer to Table 5. As the results were not statistically significant, the use of a post-hoc procedure was not employed.  Table 6 shows the Pearson correlation coefficients among the dependent measures in the study. Scores from the ILA achievement and the ILSES scores showed a moderate, positive correlation at r = 0.334 (p < .01). The highest correlation detected was between the ILSES and the LSS with a moderate, positive relationship at r = 0.392 (p < .01). The association detected between the learner achievement and learner satisfaction scores resulted a small, positive correction at r = 0.155 (p = .014).

Discussion
Prior to interpreting these findings, we believe it is important to highlight the overall limitations and delimitations of the present study. First, this study was conducted in a controlled laboratory setting with students enrolled in education courses in a public research university in the southeastern United States. Caution should be observed in generalising these findings to all types of learners (e.g., K-12) and to other more ecologically valid settings (such as completing an assignment for a course) in which the learners have different motivations to complete the task. Our participants were not necessarily highly motivated to complete the study since there were no incentives for participating. Since we have strong evidence of the relationship among metacognition and motivational constructs, it is likely this limitation shaped the results of our study (Valencia-Vallejo et al., 2019). While we used established and systematic instructional design procedures (backwards design) to create the OER, this data collection was limited to our first pilot OER participants. Thus, it is possible that revisions and further refinements would have resulted in more robust learning resources.
Regarding the learning resources, this was a reading based OER in which the baseline form of the instruction was simply the three narratives for accessing, evaluating and communicating information. Aside from these narratives, the instruction did not include application tasks or other aspects of instruction (such as multimedia resources) that could have contributed to a more robust instructional experience. This intervention should be considered an instructional resource to help gain important insights regarding the instructional elements of interest in the context of an OER on IL, as opposed to comprehensive learning materials on the topic of IL. Finally, while the ILA was pilot tested with a small group of students prior to the study and the internal consistency reliability coefficient was above the social science standard, we acknowledge that this assessment was a homegrown assessment and not a standardised measure with strong validity evidence. In light of these things, we do believe we have some worthwhile findings to discuss.
Although the study findings showed no significant differences across all dependent measures, the descriptive statistics do provide some notable discoveries. The ILA test was dichotomously scored out of a total of 36 multiple choice items. The average score across all participants was M = 20.94 (SD = 5.66), suggesting that learners had scored an accuracy of 58%, which is substantially higher than the possibility of guessing (25%) on the four distractor multiple choice items. Though we did not employ a pre-test in the procedures to control for prior knowledge (controlled by the use of random assignment), these findings do suggest the OER had an influence on the participants learning outcomes. This suggests that the target populationundergraduate students in higher educationwere able to learn IL knowledge and skills in an online OER. This finding lends credence to our approach to instruct IL knowledge and skills in an ecosystem in which they will use these knowledge and skills: the online environments (Allen, 2008), and aligns with overachieving goal of enhancing students' IL knowledge and skills.
When dissecting the participants' achievement across the domains of access, evaluation and communicate, we also observe that the participants scored highest in the area of access (approximately 62%) as opposed to evaluate at approximately 58% and communicate at approximately 54%. This finding suggests students in higher education are more comfortable with accessing information online, which includes tasks like performing searches using both search engines and research databases while discriminating between sources of information (e.g., scholarly research versus a blog). While the scores from the access sub-construct still leave room for improvement, it is worthwhile to know the students in higher education appear to be stronger in this area. Conversely, the scores from the evaluate and communicate domains were lower than that of the access domain. We know from prior research that evaluating information discovered online is increasingly challenging (Wineburg & McGrew, 2018). This is due to various factors, such as the presence of "cloaked websites" (Daniels, 2009, p. 660), algorithms and their influence on creating filter bubbles (Bastick, 2021), common misinformation techniques such as using emotional language (Basol et al., 2020) and a growing post-truth sentiment (Lewandowsky et al., 2017). Being able to evaluate information online requires knowledge and skills to discriminate between facts and opinions, including things like author bias, misleading information and factual information. Communicate, with the lowest overall score, is especially important with the advent of social media tools and their common use among undergraduate students. The communication domain addressed issues of digital footprints and identity in relation to sharing information online, how that information should be presented, and how control of such information is indeterminate once placed on the Internet.
Unfortunately, we were unable to pinpoint which instructional elements should be integrated into an OER on the topic of IL since the differences across our six conditions did not result in any discernable results across our three dependent measures of interest: achievement, learner satisfaction and IL self-efficacy. Our use of tooltip text, embedded practice, learning objectives and summaries are not entirely new innovations in the research literature. In fact, the use of learning objectives in creating different environments has been examined and refined in literature dating back to the 1960s (Mager, 1962). It is believed that stating learning objectives at the beginning of instruction will assist the learner in structuring their own learning experience (Ausbel, 1968) and enhance the forethought phase of self-regulation. Likewise, the use of summaries to influences learning outcomes has been thoroughly examined and is intended to help the learners focus on the salient information in an instructional text (Hartley & Davies, 1976), while providing a self-evaluative cycle on their mastery of the learning outcomes.
Meanwhile, the advent of tooltip text (sometimes referred to as leads) to show definitions of key terms and embedded practice questions with feedback are newer innovations that emerged and evolved with information and communication technology used for delivery (Antonenko & Niederhauser, 2010). While embedded practice has evolved with technology-enhanced learning solutions (including educational games), the principle of practice can be traced to effective instructional events based on theories of human learning (Gagne, 1985). Practice with feedback has been shown to have statistically significant effects on learning outcomes (Hannafin, 1987). Tooltip text (or leads) of the definitions of key vocabulary terms can serve as an advanced organiser in which the tooltip text can orient and prepare the reader for information presented in the expository form (Antonenko, & Niederhauser, 2010).
The fact that none of these instructional elements employed displayed a significant impact on the outcome measures (nor the full version of the online OER with all instructional elements integrated) raises some important questions for future research on the application of instructional elements to online OERs. OERs are believed to offer several instructional and technological affordances unavailable in print books and other traditional forms of learning (Fenwick et al., 2013). While instructional elements like learning objectives and review summaries are commonly integrated into regularly used print textbooks, the use of embedded questions with immediate feedback and the use of tooltip text to provide definitions of key terms are technological affordances not readily available in print textbooks. This research raises the question of which instructional elements are essential for the creation of OERs, particularly when the content of the resource is built and delivered online in the ecosystem in which the students will later practice their learning. While we achieved a sufficient sample size for our research objective, we do not believe these findings are conclusive on the application of these instructional elements integrated into OER solutions. We are calling for more research on innovative instructional elements that can be seamlessly integrated into an OER configurations online, including such affordances like text-to-speech, highlighting and annotating, graphics and animations, gamification strategies and searching utilities to name a few. The approach taken in the present study is a sound methodological approach to isolate the effects of the individual instructional elements. More rigorous empirical research is necessary, and like a lot of empirical research, this study has generated more questions than answers.
Notably, it would appear that the participants were generally satisfied with the learning experience as evidenced by the overall LSS score at M = 3.78 (SD = 0.74). As a reminder, the LSS employed is on a five-point semantic differential scale (Ritzhaupt, 2019). The participants' satisfaction with the learning materials suggests that the instructional solution did not result in an unpleasant or difficult learning experience. This is important since teaching IL knowledge and skills in an OER takes place in a similar ecosystem (the online environment) in which they will apply their learning to real-world source materials. Also notable, the students had overall high levels of IL self-efficacy, which suggests the learners believe they can apply the knowledge and skills to making informed decisions about the access, evaluation and communication of online resources in the digital environment. While their performance on the ILA left room for improvement, the participants generally believed they could make many decisions surrounding IL. The gap between their actual IL learning achievement and their self-efficacy is an area worthy of future research. Our results from the correlation analyses show a moderate, positive relationship between achievement and self-efficacy. Although this relationship is statistically significant, we would have anticipated a strong relationship between these two constructs as prior research has shown more robust relationships in other disciplines (Ayotola, & Adedeji, 2009). However, some findings on information and communication technology literacy achievement and self-efficacy also show the small to moderate relationship (Rohatgi et al., 2016). Again, we believe these findings are a call for more research on this relationship to address a gap.
We hope this article will encourage more researchers to carefully design, develop and explore the established and innovative instructional elements that are natural affordances to online OERs. In the present study, we attempted to use an OER as a natural and ecologically valid space for undergraduate students to learn and practice IL knowledge and skills, while addressing the larger concern of self-regulated learning. It was also an attractive tool because it can be implemented in instruction without any training from possible educators, or major time and effort investments among the undergraduate students. Though our findings did not produce significant differences across our treatment conditions, we believe the research has provided some insightful findings and useful avenues for future research in the domain of IL and the affordances provided by OER in the 21st century. Our hope is that this article is a call to future researchers and educators to explore this emerging concept of OERs to areas that would most benefit from their application domain, such as IL.

Recommendations for future research and creating OERs in IL
Though the presence or absence of the instructional elements under investigation in the present study did not appear to make a meaningful difference in the results, we remind our readers that the motivations of our sample may have clouded our results since this was a lab-controlled study. We believe it will become necessary for authors and developers of OER resources to employ a scientific approach to the choice of affordances integrated into these resources. Recent technological innovations (such as gamification) and testing procedures like A/B testing (Kohavi & Longbotham, 2016) can enable authors and developers to test a range of instructional elements in their OER implementations to discern which affordances offer the most contribution to student learning outcomes using the principles of the scientific method. Additionally, we recommend that future research on the efficacy of instructional elements in OERs includes a qualitative component to capture insights that may not be discovered through quantitative methods. For example, adding qualitative techniques such as interviews or focus groups to this study could have provided further nuanced insights regarding how these elements impacted learners and why meaningful differences between conditions were not found.
Authors and developers should push online OERs into new horizons by implementing novel instructional elements and blending design approaches into comprehensive products to ultimately reduce the cost of educational materials among students, provide more quality options for educators and lead to enhanced student learning outcomes across content domains. We believe that a socially relevant testbed for these new OER configurations is the domain of IL, as this skillset is needed across disciplines in 21 st century learning environments. Building effective online OERs to develop IL knowledge and skills is a natural progression that is in dire need of engagement from authors and developers in K-12 and higher education, since the students who will use these resources will transfer their knowledge and skills within the same ecosystems. The future exploration of OERs in the domain of IL is a fruitful and impactful area of research.
Note: At the time of carrying out the research the lead author, Max Sommer, was working as a PhD candidate in the School of Teaching and Learning, College of Education, University of Florida.