MORE TROUBLE FOR JOHN LOTT? In the latest installment of a back-and-forth debate over John Lott’s more-guns-less-crime hypothesis, Ian Ayres and John Donohue accuse John Lott of making systematic errors in support of his hypothesis. (Pp. 1397-98; see also 1392-93). By “systematic” errors, I mean that, according to Ayres and Donohue, the errors appear to all produce results that support Lott’s hypothesis. They don’t accuse him of fraud, and make a point of noting that it’s very easy to make mistakes in this area, but nonetheless the language is, by academic standards, rather strong. They also note (1374) that Lott removed his name from a response that relied on the data in question. (This seems odd to me — if he removed his name because the data were bad, why would the coauthors go ahead? I have no idea, and no knowledge of what’s going on there beyond what Ayres and Donohue say.)

This sort of statistical analysis is beyond my competence, and I’d really like to see what people who know more than I do have to say, but as I read this article it looks to be a substantial blow to Lott’s position. I hope that someone will check the analysis against Lott’s data and see if the coding errors that Ayres and Donohue point out are present. If they are, well, it certainly undercuts the thesis of Lott’s book, at the very least.

Here’s the original article by Ayres and Donohue, and here’s the response by Plassman and Whitley (which Ayres and Donohue say Lott removed his name from; here’s what seems to be an earlier version of the same paper with Lott as the first coauthor). The piece linked at the top is Ayres and Donohue’s reply to the Plassman and Whitley response.

By way of full disclosure, I went to law school with Ayres and Donohue, and regard them both as honest, straight-up guys notwithstanding that they have a political position that in many cases would be different from mine. Unlike some of Lott’s other critics, these guys are real scholars, writing in the Stanford Law Review, which gives their criticism considerable weight. I am, however, entirely incompetent to judge the underlying dispute on its merits, and hope that people who have the relevant expertise will weigh in.

UPDATE: Tim Lambert has a lengthy post on this now, too. One thing I’m not clear on: I believe that these are all new data, not the data presented in Lott’s earlier book “More Guns, Less Crime.” Lambert’s post implies otherwise, but it’s not entirely clear, and looking at my post above, I guess I imply the same. But unless I’m mistaken, the data in question are all new. (It’s the “more guns less crime” hypothesis in question here, not the book of the same title.) Lambert also has this post on a page he’s set up to deal with a different Lott issue — the 1997 survey that Lott was accused of not performing. It’s worth noting that this new Ayres/Donohue issue is a distinct question, with as far as I know no connection to the 1997 survey issue beyond the author. And it’s a question that has a direct substantive relationship with the core of Lott’s scholarship in a way that the survey issue, or the Internet pseudonym issue, did not.

UPDATE: John Lott sends a lengthy email, of which this is the meat:

When I agreed to do the paper for the Stanford Law Review that responded to Ayres and Donohue’s attack on my work, I got a promise both verbally and in writing that “any reaction to your responses won’t be incorporated into their article, but rather will be part of their reply” (see e-mail below dated August 29, 2002 from Ben Horwich). However, after we completed our piece, Ayres and Donohue insisted on making changes to their original paper. Initially, I declined letting them make the change. The Stanford Law Review (due to pressure from the authors) would not take “no” for an answer. In response, I offered a compromise where we could make one change in exchange for allowing them to make their change. That was turned down. We were then given an ultimatum where we either agree to the change being made or Ayres and Donohue’s paper would be published without ours. (Not surprisingly, the issue of a new change for Ayres and Donohue was revisited yet again when Plassmann and Whitley dealt with the final galleys, but I don’t have the e-mails on this.)

I talked with Jeff Parker at George Mason University about this and he suggested that we withdraw the paper from the review and send it someplace else. That seemed fine with me, but I knew that my younger co-authors would be more risk averse and also wanted the Stanford Law Review’s name. As a second alternative, Jeff suggested that I withdraw my name from the piece and hopefully use it as leverage to get the editors to do the right thing. After communicating with my co-authors that is the response that we agreed to take, and I thought that would be the end of the story.

However, the Stanford Law Review allowed Ayres and Donohue to add an addition to their piece commenting on all this.

There’s more, but that’s the gist, I think. Lott adds: “Just for the record, I still believe that ‘Ayres and Donoue have simply misread their own results.'”

ANOTHER UPDATE: Chris Lawrence compares John Lott and Paul Krugman. He has more thoughts here.

YET ANOTHER UPDATE: Tim Lambert has posted an email from the Stanford Law Review. On the one hand, it makes clear that Lott didn’t take his name off the article because of the data-coding issue. On the other hand, it seems that the data-coding issue still isn’t resolved. There’s an article (subscription require) in the Chronicle of Higher Education on this. Here’s the most important bit:

In the years since Mr. Lott’s first publication, at least six scholars have published studies that tend to confirm his findings, while at least four other studies have tended to cast doubt on his findings. Mr. Donohue noted in an interview that Mr. Lott’s research has convinced his peers of at least one point: No scholars now claim that legalizing concealed weapons causes a major increase in crime. Even Mr. Donohue’s analysis, which is highly critical of Mr. Lott’s, finds only “modest pernicious effects,” in his words.

Mr. Lott’s 1997 paper on gun policy was, “to that point, the most important piece of empirical research that has ever been done in the social sciences,” says Jeffrey S. Parker, a professor of law at George Mason University. “I doubt that even Ayres and Donohue would dispute that point.”

Mr. Ayres and Mr. Donohue’s critique of Mr. Lott’s scholarship runs as follows: The models used by Mr. Lott and his co-authors have not taken sufficient account of the broad differences between states that permit the concealed carrying of guns and those that do not.

In particular, Mr. Ayres and Mr. Donohue suggest that the spike in murders associated with the crack-cocaine epidemic of the late 1980s was concentrated in urban areas in states with restrictive gun laws, while states that permitted people to carry concealed weapons in the 1980s tended to be relatively
rural and unaffected by drug violence. That imbalance, Mr. Ayres and Mr. Donohue say, has given the right-to-carry states an artificial boost in studies by Mr. Lott and his allies.

In their reply, Mr. Plassmann and Mr. Whitley argue that their opponents’ own data, when properly read, demonstrate immediate state-level benefits from the legalization of concealed weapons. They also present new county-level data for the period 1977-2000, which they say further supports the more-guns, less-crime thesis, whether one uses their opponents’ preferred statistical techniques or their own.

It is here, in this new 1977-2000 data set, that Mr. Ayres and Mr. Donohue claim to have identified a serious set of coding errors. Mr. Plassmann and Mr. Whitley failed to assign dummy variables (which researchers use as place holders, to stand for meaningful variables that they may have neglected to
identify) for the states of Alaska, Massachusetts, and Pennsylvania for certain years in their calculations.

Correcting those errors, Mr. Ayres and Mr. Donohue write, “completely reverse[s]” the paper’s conclusions and “restore[s] the conclusion that concealed-carry laws were associated with increases in crime (or no effect) for all crime categories.” . . .

Mr. Lott replies that the alleged coding errors are irrelevant to the larger debate. “Whether one believes the regressions in the Plassmann and Whitley piece or not, just looking at Ayres and Donohue’s own results — you can’t look at the graphs that Plassmann and Whitley have of Ayres and Donohue’s results and not see a significant drop in violent crime.”

“The basic results are not fragile,” Mr. Whitley writes in an e-mail message. “Minor errors in coding would not undermine them (and an entire literature).” Mr. Whitley says that he could not reply to the charges in detail because he had not yet had time to carefully review Mr. Ayres and Mr. Donohue’s comments. Because the allegations appear in a law review rather than a peer-reviewed academic journal, no third-party scholars have reviewed the claim of coding errors.

Six tables that derive from the same allegedly miscoded data set appear in Mr. Lott’s new book, The Bias Against Guns: Why Almost Everything You’ve Heard About Gun Control Is Wrong (Regnery, 2003). James Lindgren, a professor of law at Northwestern University, says, “If Donohue and Ayres’s account is as it appears — and I’m not in a position to judge that — then Lott should withdraw the book for revision.”

Mr. Lindgren adds that he believes it extremely unlikely that any coding errors were the result of a conscious intent to distort the study’s findings. He notes that Mr. Lott has not only shared his data sets with other scholars, but has made them generally available to the public on his Web site. “You tend not to do that if you’ve intentionally miscoded your variables,” he says.

So that’s where we stand now.