A conversation got started over on abetteroakland.com about crime in Oakland. V Smoothe (one of the truly great Oakland bloggers), wanted to make the point that crime in Oakland is bad, and to do so she used the FBI’s Preliminary Annual Uniform Crime Report to build a graph comparing the crime rates of several cities to show that Oakland’s is particularly bad. While I agree that crime is bad and that some people are unrealistic about the existence of a problem, I said that the numbers aren’t an honest or conclusive comparison of cities.
Much anger ensued, and the conversation about something very important got derailed into a discussion of how I can’t do math. I wanted to move the less important part of that conversation over here, to separate it from the important discussion.
Now, V is much smarter than I am and extremely sharp when it comes to analyzing policy issues. I’m not kidding – if you want to understand Oakland politics she is a vital source of information. But trying to rank cities based on the ratio of crimes to population isn’t a particularly good method. As the FBI notes on the page from which V got these numbers:
Individuals using these tabulations are cautioned against drawing conclusions by making direct comparisons between cities. Comparisons lead to simplistic and/or incomplete analyses that often create misleading perceptions adversely affecting communities and their residents.
I should let it go at that. But I do think it’s important to understand why a simple ratio isn’t a good tool for comparing cities with significantly different populations. I’m not sure I can explain it simply and concisely (I failed over on abetteroakland), but I want to try. Please keep in mind that there are whole books written on this specific issue. It’s hard to summarize without leaving big holes. Worse yet, I’m not a statistician, so I’m not the best person to be summarizing this stuff. I’m merely dumb enough to try.
Let’s dig in:
What’s your problem with simple ratios?
I love simple ratios. They just don’t tell you much because they’re simple.
Currently, NYC and Boise, Idaho have the same crime rate as expressed in the violent crimes committed, per 10k citizens stat (7.8 – nice job NYC and Boise). You can say fairly that in a simple ratio, the same number of people out of 10k are victims of violent crime in both of these cities. No argument there. The problem comes when you start using this stat to make a qualitative judgment like “good” or “bad”. Is crime in Boise is “as bad” as in NYC? Maybe, but that’s not borne out by the statistics.
Just having the same ratio of things is very different from equivalency. While both NYC and Boise have the same ratio of violent crime to population, there is more crime in NYC, and it has a different impact on society. Or, to put it another way, 4 out of 5 dentists agreeing that you should chew Trident is a lot different than 4,000,000 out of 5,000,000 dentists agreeing that you should chew Trident. One sample is significantly broader, and likely more representational of the opinion of dentists generally. The sample size matters. A lot.
If, for the sake of argument, there were 5 murders in Boise this year, their murder rate relative to their population would grow substantially (from 3 to 8 – the highest murder rate in the last 100 years). While the number in that ratio would have gone up dramatically, that doesn’t mean you can then say crime in Boise is worse than in NYC. There’s still more crime in NYC. Going further, the way crime affects one city is very different from another. The impact of crime isn’t quantifiable by the crime stat.
The smaller the sample size, the more dramatic the impact of a single murder. And just as significantly, the smaller the sample size, the more erratic the data. If there were 10 more murders in Boise this year and 5 fewer the next, it wouldn’t make sense to say there was a crime wave that was solved and the situation was improved based on those numbers. The sample is just too small to compare to a much larger sample.
So why not just take out the lowest population cities so they stop skewing the numbers?
Because the high number skews things as well. It takes a lot of murders for a city as large as NYC to have that ratio move, in the same way that it takes very few in Boise. Any time you use a ratio instead of a mean to make a comparison between 2 data sets, outliers will have an amplified affect. Similarly, the outside points in your sample will by definition skew the ratio disproportionately.
Why not take out both the highs and lows?
This is a better way to represent relative crime more accurately, and the narrower the band you’re looking at the more accurate the relative crime data is going to be. Even better if you weight the outside points on your sample to compensate for the amount that they will skew the data (for instance, having the ratio be stated in units of 8k at the bottom end of the sample and 12k at the top, with a bell curve in between). But I’m still not convinced that it says anything about how “good” or “bad” crime is when comparing 2 different cities. It would probably be more statistically accurate, but it still wouldn’t be qualitative.
All of this ignores a whole host of other issues that make the UCR a bad tool for ranking cities against each other. It’s based on self-reported crime stats from police, which can be manipulated for political purposes and aren’t subject to the determinations of coroners or further adjudication. It doesn’t account for the ratio of police to citizens, or the tactics utilized by the city for preventing crime. It doesn’t factor in a whole host of important demographic factors, like relative wealth and education of the citizenry, or population density. It also doesn’t provide any useful means of accounting for the way crime localizes inconsistently in a given city.
Doesn’t representing that figure per 10k of population offset the different sizes of the sample?
No. 10k is an easily expressible unit of measurement, but it’s only expressing the same simple ratio. Oakland’s violent crime rate per 10k is 98.68, but you can express the same ratio in terms of 100k of population (986.8), 1k of population (9.868), etc. All that means is that you’re moving the decimal point, not that you’re factoring in the size of the sample.
This is the standard way of expressing this data. What’s your alternative?
If you want to make the point that crime in Oakland is bad, post the stats for Oakland. Comparing the UCR numbers for Oakland over the last decade would be an extremely useful exercise, and for that usage the UCR numbers are great. And it would more effectively make the point that V was trying to make: that Oakland has a serious crime problem.
As far as ways to compare crime in cities regardless of the size of the population to determine “good” and “bad”, I don’t have one. I don’t think there is one. How do you compare anything about 2 cities that are entirely dissimilar?
Okay, why does this even matter? This seems like a minor point compared to the one we’re all trying to get outraged about.
Because when you post a list of cities and say, “crime in Oakland is worse than all of those other cities on this list”, you’re making a qualitative judgment which the stats don’t support. In the same way that people were pointing out 2 weeks ago that Oakland’s murder rate was down 60%, it’s a sensationalistic use of statistics that doesn’t actually mean much. And it’s not like you need to use misleading stats to establish that crime in Oakland is bad. It is.
It’s similar to the way people point to the Dow-Jones and say that the economy is tanking. Yes, the numbers for the Dow and the data on the general economic state of our country show that both are tanking. But one doesn’t represent the other. The Dow shows what people are willing to pay for stocks in 30 of the largest companies in the country. While that number bears some relation to the state of the economy, it doesn’t express the state of the economy. It doesn’t even express the fortunes of the 30 individual companies it represents. It’s a very crude aggregate, which is useful for some things, but shouldn’t be given too much weight on economic issues.
So. I hope I’ve made the case that UCR stats are not a good metric for comparing cities. That’s not what they’re designed to do, and not what they should be used for. But if I haven’t convinced you, I hope the American Society of Criminology will. This is the wording of a resolution they passed on this very subject:
Be it resolved, that the Executive Board of the American Society of Criminology opposes the use of Uniform Crime Reports data to rank American cities as ‘dangerous’ or ‘safe’ without proper consideration of the limitations of these data. Such rankings are invalid, damaging, and irresponsible. They fail to account for the many conditions affecting crime rates, the mismeasurement of crime, large community differences in crime within cities, and the factors affecting individuals’ crime risk. City crime rankings make no one safer, but they can harm the cities they tarnish and divert attention from the individual and community characteristics that elevate crime in all cities. The American Society of Criminology urges media outlets to subject city crime rankings to scientifically sound evaluation and will make crime experts available to assist in this vital public responsibility.