Holistic Evaluation
What is holistic evaluation?
‘Holistic’
Definition of holistic as a philosophical term: “characterized by the belief that the parts of something are intimately interconnected and explicable only by reference to the whole (Oxford, 2015).”
“Holistic” is a philosophical term, examining the system as a whole rather than its constituent parts in isolation. In evaluation, the authors surveyed for this webinar considered it essential to examine the whole system in which the education technology tool is being used: to consider the environment in which it is used as well as delving into its impacts on learners and organisation (Pickering and Joynes, 2016; Pickering, Lazarus & Hallam, 2019; Mayer et al., 2014; Cook & Ellaway, 2015).
Cook & Ellaway (2014) point to the emergence of more discussion around the TEL being evaluated contextually, beyond the resource being evaluated ‘in its own right’ (p. 962). They propose questions that contextualise the learning program or resources according to why they are required, from a learner’s need through to organisational/societal needs as well as usability testing, ‘participant experience and satisfaction’ as well as documenting processes, decisions, and ‘key events during implementation and the final product’ (Cook & Ellaway, 2015 p. 966).
Through extensive research, Mayer et al., (2014) have developed a framework to evaluate ‘serious games’ (p. 502) in their context, working towards a standard comprehensive method that can be used across all games. They suggest questions for evaluation in the pre-game phase around the following classifications:
“Socio-demographics: sex, age, nationality, culture, etc. (Bekebrede et al., 2011; Brown, Ley, Evett & Standen, 2011; Jenson & de Castell, 2010; Kinzie & Joseph, 2008; Pfister, 2011).
Previous experiences/skills: with computers, games, VLEs, etc. (Erfani et al, 2010; Harper et al, 2007; Mortagy & Boghikian-whitby, 2010).
Attitudes: change, conflicts, intrinsic and extrinsic motivation, learning styles, etc. (Ashton & Lee, 2009; Garris, Ahlers & Driskell, 2002; Guay, Vallerand & Blanchard, 2000; Huang, 2011; K. Lee & Ashton, 2004).
Skills: personal competence (with games, learning, certain professional skills, etc.) (D. J. Brown et al, 2011; Enochsson et al, 2004)
Behaviors (behavioral intentions)
Group, team, organizational characteristics: team/group conflict, psychological safety, psychological collectivism, team and organizational commitment, etc. (Brockner & Higgins, 2001; Carmeli, Brueller & Dutton, 200)” (Mayer et al, 2014: p. 510)
Overview of the 4 levels of a holistic evaluation model
4 levels of Holistic Evaluation proposed by Pickering and Joynes (2016)
Technology enhanced learning (TEL) resources are tools and applications that are designed to improve the learning outcomes for students. The introduction of any educational learning tool requires justification and a review of its impact on student learning. The evaluation method proposed by Pickering and Joynes (2016) outlines four levels of evaluation that can be completed either individually or as a complete end to end evaluation from inception of the tool to its institutional benefit.
The four levels are:
Level 0 - Preliminary evaluation of learner need and Development
Level 3 - Institutional impact (not included in this learning resource)
The holistic evaluation model provides a step by step method for evaluating a TEL resource individually and as part of a whole institution. As mentioned above, justification of financial support and integration are needed to implement a TEL resource at any level of the institution and this will require data that is collected from the four levels.
The first 3 levels have been explored in greater detail. You can navigate to levels 0, 1 and 2 by clicking on the links above.
Extension Reading
Stufflebeam, D. L. (2003). The CIPP Model for Evaluation. In T. Kellaghan & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 31–62). Springer Netherlands. https://doi.org/10.1007/978-94-010-0309-4_4