Once the Technology Enabled Learning (TEL) resource is developed, it must be piloted with a small sample of learners to evaluate whether it achieves adequate levels of learner satisfaction and that there is superior learning gain compared to existing the learning resource.

Level 1a Learner satisfaction

This level requires that satisfaction, consisting of ‘attitudes, behaviours and opinions’ (Pickering, Lazarus & Hallam, 2019, p. 109), is measured in a controlled environment. This evaluation level should not venture beyond establishing whether the learner is satisfied with the aesthetics of the resource and whether they believe the content is relevant to the curriculum requirements. Gauging learner satisfaction is crucial for assessing whether the learner will engage with the resource (Pickering & Joynes, 2016).

If the satisfaction levels are evaluated as low, it is vital to find out why this is. This response should be used as feedback to the development phase to improve the resource (Pickering & Joynes, 2016).

Both quantitative and qualitative methodologies can be used to evaluate learner satisfaction. A recommended quantitative evaluation tool is a Likert-scale questionnaire (Pickering et al., 2019). Qualitative evaluative techniques include focus groups, interviews and ‘talk aloud’ investigation of usability (Chaudhuri et al., 2015, Eccles & Arsal, 2017 in Pickering, Lazarus & Hallam, 2019, p. 110). The evaluation must be designed with due consideration given to peer-reviewed literature focusing on the research questions to be addressed (Pickering, Lazarus & Hallam, 2019).

During the pilot, the survey sample will most probably be a small one. Later, when the resource is deployed across an entire cohort, the larger set of responses to the survey will contribute to a more detailed portrait of learner satisfaction (Pickering & Joynes, 2016).

Level 1b Learning gain 

If learner satisfaction is evaluated as high in Level 1a, ‘knowledge, application, or skills’ (Pickering et al., 2019, p. 1100) should be evaluated compared to that achieved by existing resources.

Calling for volunteers to answer questions both prior to and after using the resource under controlled conditions is recommended. (Issa et al., 2011). It is suggested by Pickering (2015) that a randomised controlled trial (RCT) could be used to carry out the testing with one group learning via existing resources and the other group using the TEL resource. To confirm retention of knowledge over time, post test questions should be given over time. Different question types will assess different levels of learning (Pickering & Joynes, 2016).

Pickering et al., (2019) remind the reader that volunteers must be recruited in an ethically appropriate manner, e.g. the time taken to complete pre and post testing does not impact on the volunteers’ other studies and lesson times.

The results of the tests should be examined according to learner demographics such as gender and age (Pickering & Joynes, 2016).