Overview
I tested three widely repeated claims about crime in the UK against the full police record of 65 million crimes. The dataset was too large for spreadsheet tools, so I built the pipeline in Apache Spark, tested it on a sample, and ran it on the full data on an Azure virtual machine.
The questions
- Are burglary rates steadily increasing over time?
- Does Liverpool (Merseyside) have the highest rate of firearms incidents in the UK?
- Are firearms offences strongly linked to drug-related crime?
Datasets
- Police crime records: UK Home Office street-level crime data from data.police.uk, 65,078,250 records with monthly data from December 2010 to May 2021.
- Population: Office for National Statistics population figures for 34,753 small areas (LSOAs), from the 2011 Census.
The raw data isn't in the repository because of its size; both sources are publicly available.
Tools
Apache Spark (PySpark) on a Microsoft Azure virtual machine, Spark MLlib for correlation and regression, statsmodels for the stationarity test and SARIMA forecast, and pandas, Matplotlib and Seaborn in a Jupyter notebook.
Data preparation and method
I loaded the crime data into Spark with an explicit schema, checked the columns and date range, and listed the 16 crime categories. I kept only the three I needed: Burglary, Possession of weapons and Drugs.
Claim 1: burglary over time
I counted burglaries per month and plotted them with a 12-month rolling average. An augmented Dickey–Fuller test showed the series wasn't stationary, so I tried detrending and seasonal differencing before fitting a SARIMA model, chosen by grid search on the lowest AIC (123.29). The model was trained on data up to December 2019 and used to forecast 2020.
Claim 2: firearms by area
The data doesn't record which weapon was involved. As the closest available measure, I counted Possession of weapons offences where the offender was sent to prison (10,416 of 283,189 records), on the basis that illegal firearms are more likely than other weapons to lead to a prison sentence. I joined the small-area population figures to each police force and calculated a rate per 100,000 people, so large and small forces could be compared fairly.
Claim 3: weapons and drugs
For each of 45 police force areas I counted all Possession of weapons and all Drugs offences, then measured the relationship with Pearson and Spearman correlation in Spark MLlib and fitted a linear regression on an 80/20 train–test split.
Key findings
Burglary fell from about 42,000 a month in 2011 to about 20,000 a month in 2021, roughly half, with a sharp drop after the first COVID-19 lockdown in March 2020. The forecast didn't point to a rise either.
Merseyside ranked 5th, at 22.3 incidents per 100,000 people. West Yorkshire was highest at 23.3, followed by Humberside, Cleveland and Northumbria. The top five are within about one incident per 100,000 of each other, so the ranking is close.
Across 45 force areas, weapons and drug offence counts were strongly related: Pearson 0.917 and Spearman 0.883. The regression predicted about 0.10 more weapons offences for each additional drug offence. This shows the two rise and fall together; it doesn't show that one causes the other.
Why this matters
Claims like these shape where attention and resources go. Checking them against the full record, and comparing areas by rate rather than raw totals, is the kind of evidence a decision-maker needs before acting on a headline.
Limitations and what I learned
- The firearms measure is a stand-in. Weapon type isn't recorded, so prison outcomes for Possession of weapons will include some non-firearm cases and miss firearms cases still awaiting an outcome.
- Population figures come from the 2011 Census while the crime data runs to 2021, so the rates per 100,000 are approximate.
- Claim 3 compares raw counts, not rates. Larger forces record more of every crime, so force size explains part of the correlation, and London is a strong outlier.
- None of the transformations fully passed the stationarity test, so the forecast should be read as supporting evidence rather than a precise prediction.
- Drug and weapons offences are mostly recorded when police find them, so these figures partly reflect police activity, not only how much crime happens.
- The data includes crime locations, which raises privacy concerns; anonymising locations to the nearest road would reduce that risk.
The biggest lessons were to normalise by population before comparing areas, to be explicit when a measure is only an approximation, and to keep correlation and causation separate when writing up a result.