Validation of a novel evaluation framework by examination of an audio-visual intervention designed to improve junior doctors’ management of the feverish child

A large amount of resource is expended attempting to change clinical practice; however the effects of these interventions are seldom assessed. A systematic literature review revealed that existing outcome measures to examine the effects of a ‘practice changing intervention’ (patient video cases as the instructional element in an e-learning package) lacked validity. As existing frameworks for the evaluation of practice changing interventions may not adequately determine an interventions’ true effectiveness a novel “7Is framework” was developed. This contains 7 domains: Interaction, Interface, Instruction, Ideation, Integration, Implementation and Improvement. To produce bespoke outcome measures for the “7Is Framework” domains the following studies were performed: i) Focus groups with junior doctors demonstrated poor differentiation between concepts of competence and confidence. A new indicator of self-awareness, the perception of “safeness”, was generated (Ideation). ii) Paediatricians rated quality and acuity of a selection of clips at differing technical qualities to produce gold standard responses for video case questions (Integration). iii) A novel matrix determining how experience and tacit knowledge may impact on patient outcomes was validated as a measure of behaviour change (Integration). iv) A time series of presentation and admission rates of feverish children provided outcome data for the evaluation (Implementation / Improvement). An audio-visual intervention in paediatric fever was designed, delivered and tested against the new system (studying 202 UK doctors). Interaction with the intervention was variable, only 28.7% completed the post learning section and issues were identified with accessing the video cases. Although measures of ideation increased and there was a trend towards behaviour change, full implementation of the guidance did not occur and overall admission rates increased. The 7Is Framework allows the various effects of an intervention to be conceptualised, promoting the development of a set of valid and specific outcome measures, ultimately leading to more robust evaluation.