This analysis of police misconduct complaints data was conducted for the May 28, 2021, Financial Times story "Small share of police draw third of complaints in big US cities". Read the findings and the computer code that drove the analysis at analysis.ipynb.
The data for this analysis came from:
-
The Invisible Institute (Chicago): The Invisible Institute filed a series of FOIA requests to the Chicago Police Department as part of its Citizen Police Data Project. The FT downloaded the datasets from the Invisible Institute's GitHub repo, and saved them to the
input/chicagodirectory. The code used to load and pre-process the data can be found atcleaning/chicago.ipynb. -
The New York Civil Liberties Union (NYC): The NYCLU filed a series of FOIA requests to NYC's Civilian Complaint Review Board to obtain its NYPD Misconduct Complaint Database. The FT downloaded the dataset from the NYCLU's GitHub repo, and saved it to the
input/nycdirectory. The code used to load and pre-process the data can be found atcleaning/nyc.ipynb. -
OpenDataPhilly and Sam Learner (Philadelphia): The Philadelphia Police Department releases data publicly on a trailing 5-year basis, meaning older data is overwritten by newly-released data. Data for 2016 onwards is from OpenDataPhilly, the city's open data website, and data for 2015 is from Sam Learner, who previously collected it for another project. The FT downloaded and saved the data to the
input/phillydirectory. The code used to load and pre-process the data can be found atcleaning/philly.ipynb.
The cities' data cannot be directly compared because each municipality records and investigates misconduct complaints differently, and the scope of the data obtained for each city is different. Below is a tabular representation of the differences in the raw data from each city:
| City | Most recent complaint recorded | Includes pending complaints | Includes complaints investigated by Internal Affairs | Includes complaints made by other officers | Multiple officers can be named under a single complaint | The same officer be named in multiple allegations under a single complaint | Each row in the data is a separate allegation |
|---|---|---|---|---|---|---|---|
| Chicago | March 2018 | x | x | x | x | x | |
| NYC | February 2021 | x | x | x | |||
| Philadelphia | March 2021 | x | n/a* | x | x | x |
*In Philadelphia, the vast majority of citizen complaints are investigated by the police department's Internal Affairs Bureau rather than a civilian review agency. The data we use for Philadelphia are IAB-investigated complaints.
For more information, see the "Notes on the raw data" section in the notebooks for each city in the cleaning/ directory. The cleaned data are saved in the output/ directory.
The FT's calculations for this story were done in R and are available at analysis.ipynb.
All code in this repository is available under the GNU General Public License v3.0. The data in the input/ directory has been made public by the entities described above.