Judging demands and predicting task difficulty: A systematic review of the literature

Funder: Oxford University Department of Education (£3,366), 1 April – 31 October 2016; Principal Investigator: Dr. Yasmine El Masri; Advisors: Professor Jo-Anne Baird, Professor Pauline Rea-Dickins; Research Assistant: Kate Cantrell.

Dr Yasmine El Masri has been awarded funding to conduct a systematic review on judging demands and predicting task difficulty.

Predicting the difficulty of tasks accurately is of prime importance in education at pedagogic and testing levels. Teachers need to match classroom activities and assignments with their students’ abilities. Similarly, item writers need to manipulate the construct assessed to target specific performance levels identified in the curriculum. Moreover, being able to gauge item difficulty accurately diminishes threats to item security and reduces test development costs due to field trialling with large samples.

Many empirical studies have investigated features influencing question difficulty and demands. Nonetheless, predicting item difficulty accurately remains a major challenge in educational assessment for both teachers and test designers. Empirical attempts have not yet exceeded 25% of variance explained.

This project aims to produce a systematic review of the literature on predicting item difficulty and judging task demands. It gathers seminal empirical work carried out in this area since 1980 and draws on various research fields (e.g. educational research, educational assessment and measurement, language assessment, cognitive psychology). The systematic review is intended to highlight weaknesses and strengths of methodologies applied in earlier studies and to set a research agenda for a future research programme.