Reflections on using administrative data to evaluate justice services
Categories: Blogs, ADR UK Research Fellows, ADR England, Crime & justice
15 March 2024
Dr Georgina Mathlin is undertaking a Cabinet Office-funded Evaluation Fellowship at the Ministry of Justice (MoJ), a jointly led opportunity by the Economic and Social Research Council and MoJ. This blog reflects on her experiences of working with administrative datasets made available through the MoJ Data First programme (funded by ADR UK).
There is a vast amount of administrative data collected across the justice system, which has been made available through the MoJ Data First programme. Being seconded into the Data and Analysis team within MoJ meant I was able to gain access to this data early in the fellowship, developing my familiarity with and understanding of the data structures quickly. This has meant I was able to get stuck into exploring the datasets for interesting and priority research questions straight away.
The fellowships structure and focus
In the first phase of my fellowship, I identified priority evidence gaps from the MoJ Areas of Research Interest that could be addressed using the administrative justice system datasets, and defined my methodology. I am now in the implementation phase: I am leading on two evaluations and have spent the past 12 months working with the datasets.
The first project evaluates the use of combined punitive and rehabilitative requirement orders for people under probation supervision in comparison to the use of either requirement order in isolation. Qualitative work undertaken by the MoJ Data and Analysis team has suggested that the combination of a punitive and rehabilitative requirement may counter-act one another and impact upon requirement completion outcomes. This study is therefore exploring the outcomes of different requirement order combinations to help advise how and when combined orders may be most effectively used. This will support policy insights on what can help reduce reoffending.
My other project is evaluating the impact of ‘Release on Temporary Licence’ (ROTL) 2019 policy changes on re-entry to the criminal justice system. Despite ROTL being a well-established practice within the open prison estate, little research has explored whether the intended effect of reducing reoffending in the long-term and increasing rehabilitation into the community is being achieved. This study therefore addresses an evidence gap into a common practice within the open prison estate to ensure ROTL policy is implemented as intended and has the desired outcomes.
Data linkage and processing takes time
I have an interest in understanding psychosocial determinates (e.g., mental health, housing, employment) of criminal justice system outcomes. Much of this data (which is mainly held in an offender assessment system) is not currently available to be linked to the other datasets – although there are plans for this to be available in the future. This means the projects are currently missing the inclusion of key variables, which may impact outcomes in the analysis. The projects can therefore be considered as sound starting points to build upon when further data becomes available.
One of the key benefits of Data First is having the ability to link datasets together, enabling us to understand offender journeys through the criminal justice system. But the practical side of linking the datasets has been a challenge. Often individuals have many ‘events’ within the datasets (e.g., for multiple offences). This can make joining the datasets together difficult in terms of ensuring the correct ‘event level’ information is being combined, for example information from one court outcome is joined with details about another, which provides inaccurate information.
Any research which uses data that was not initially collected for research purposes requires processing to ensure it is structured in a ‘research-ready’ state. This is a time-consuming process and has been something I have spent a considerable amount of time doing during my fellowship. I have also spent lots of time exploring and understanding the missing data, which can bias results. For example, an underlying pattern to the missing data (for instance, where certain groups of people are more likely to have missing data) can mean the results obtained are not generalisable to the wider population.
Final reflections
Despite some of these challenges, I have found the fellowship has developed my data analysis understanding and skills, especially from working alongside colleagues within the Data and Analysis team and learning from their approach to the analysis process. Working alongside policy-facing analysts has also ensured my current research has a clear relevance to policy. It is exciting to see that projects I have been working on over the past year have begun to yield results, and to know that they are directly reaching policymakers so my research can have real impact..