Secondary school accountability in England

This research used data made available via the Office for National Statistics (ONS) Secure Research Service, which is being expanded and improved with ADR UK funding.

Authors: Lucy Prior, George Leckie (University of Bristol), John Jerrim (University College London Institute of Education) and Dave Thomson (FFT Education)

Date: June 2021

Research summary

This work presents a comprehensive statistical evaluation of the Progress 8 school accountability measure in England, identifying its strengths and weaknesses. This enabled the researchers to make six recommendations to policy makers.

Progress 8 is a secondary school accountability measure for state-funded schools first implemented in 2016. Its findings have important consequences for schools and pupils as they are published in school performance tables to hold schools accountable and help parents choose a school. They play a central role in national discussions about the state of education in England.

Due to Covid-19, GCSE and Key Stage 2 tests were cancelled and the government chose not to calculate Progress 8 for 2020 and 2021. This created an opportunity to investigate the statistical challenges, as well as the broader long-standing concerns, for school accountability.

Data used

This study used the National Pupil Database (NPD) made available by the Department for Education (DfE) through the Office for National Statistics (ONS) Secure Research Service (SRS). The NPD covers all state-educated pupils within English schools. Variables include age 16 GCSE exam results alongside anonymised demographic and protected characteristics data.

This study also used publicly available school performance data from the Department for Education.

Methods used

This study followed a three-step process. First, Progress 8 was reviewed in terms of the calculation, presentation and interpretation in school performance tables.

Researchers then assessed the suitability of Attainment 8 as the output measure at Key Stage 4. This is calculated by allocating points to each student based on the grade in up to eight subjects and from which Progress 8 scores are derived. Progress 8 is a value-added measure which compares Attainment 8 scores for pupils with similar results at Key Stage 2 (KS2, age 11). This led to a review of KS2 scores as the sole input variable to Progress 8, stimulating the debate on the inclusion of pupil background characteristics such as gender and disadvantage, building on earlier work under the same wider project.

The next step looked at several issues surrounding Progress 8, including statistical strengths and weaknesses relating to:

• decisions around which schools and pupils are excluded from the measure
• presentation of Progress 8 to users, choice of statistical model, and calculation of statistical uncertainty
• issues related to the volatility of school performance over time, including scope for reporting multi-year averages
• challenges for Progress 8 raised by the Covid-19 pandemic.

Research findings

This study identified the strengths and weaknesses of Progress 8, with six recommendations to improve it:

Progress 8 strengths:

• accounting for school differences in prior attainment at intake
• encouraging focus on students across the distribution of performances rather than just at GCSE grade C/D boundary
• presenting statistical uncertainty through 95% confidence intervals and colour-coded bandings.

Progress 8 weaknesses:

• increased pressures on schools and pupils by excessive emphasis on English Baccalaureate (EBacc) subjects as these are heavily favoured in Attainment 8 scores
• biases introduced through a lack of sufficient contextualisation for variation in pupil backgrounds across schools
• reduced accountability for mobile pupils (children who join and/or leave school outside the normal age of finishing education)
• the lack of understandable difference between schools
• the instability of Progress 8 scores over time and its consequences for parental school choice.

Six recommendations:

1. Present a less EBacc-focused measure alongside Progress 8 to provide a holistic picture of school performance relevant to a greater number of schools and pupils
2. Present a pupil background adjusted measure alongside Progress 8 to provide a picture of school performance informed by school context
3. Recognise pupil mobility by making school Progress 8 scores an average of all pupils who attended each school, weighted by their time in each school
4. Communicate more clearly the relative importance of school Progress 8 scores in explaining the overall variation in pupil progress and the magnitude of each school’s individual Progress 8 score
5. Increase warnings regarding the substantial uncertainty in using Progress 8 to predict the future performance of schools
6. Report multi-year averages for Progress 8 alongside current single-year summaries to illustrate and counter the instability of school performance over time.

These findings acknowledge that school performance measures offer very little to inform strategies for school improvement. They also highlight issues related to the high-stakes accountability system of Progress 8 that produces harmful consequences such as gaming behaviour and perverse incentives. For example, directing students away from certain subjects in case they fail or making decisions based on improving their position on league tables rather than the needs and welfare of students. The above recommendations aim to alleviate these issues.

Research impact

This study is one output from a larger project on school performance measures led by George Leckie and which has received £379,433 of funding from the UK Economic and Social Research Council (ESRC).

In joint work with the Northern Powerhouse Partnership, this project produced a website where schools could see their Progress 8 and adjusted Progress 8 scores. The adjusted scores consider prior attainment and other pupil characteristics, such as socioeconomic status. This was accompanied by media interest and sparked further interest from those involved in the school sector.

This project has been presented in several advisory committees at local, national and international level.

Research outputs

Journal articles

• Review of Education article, June 2021: A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses and challenges for ‘Progress 8’ raised by COVID‐19
• British Educational Research Journal article, 2019: The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England.

Other publications and reports

• Fairer Schools Index, Northern Powerhouse Partnership, 2019: Fairer Schools Index
• University of Bristol, Policy briefing, January 2019: The Government’s Progress 8 school performance measure needs to account for pupil background
• British Educational Research Association Blog, 2019: School accountability by progress 8: A critique of Conservative and Labour proposals
• British Educational Research Association blog, October 2021: Is it still worth calculating Progress 8?
• University of Bristol, Policy Briefing, 2021: Is it worth calculating Progress 8 anymore?

Presentations, blogs and news posts

• Accountability and school differential effects, talk at University College London, 2018
• Heart Radio, pre-recorded interview, January 2019
• Love Sport Radio, live interview January 2019
• Presentation of Progress 8 research to MPs at private meeting with UK Parliament Education Select Committee, March 2019
• National Education Union annual conference, Liverpool, April 2019
• British Educational Research Association, recorded seminar on ‘The importance of adjusting for pupil sociodemographic in Progress 8 and school accountability in England’, 2020
• National Pupil Database User Group Meeting, London, January 2020

About the ONS Secure Research Service

The ONS Secure Reseach Service (SRS) is an accredited trusted research environment, using the Five Safes Framework to provide secure access to de-identified, unpublished data. If you would like to discuss writing a future case study with us, please get in touch: IDS.impact@ons.gov.uk.