Whether your tastes are those of Margaret Atwood or Robert Heinlein, having a large pastorate and giving divinely inspired guidance on the direction of a community would seem to make a great futuristic novel.
But, it seems we’ve been living through one for years. Never run by a literal spiritual religion, but by a symbolic one: economists. All the analysis, tips, and suggestions they churn out for stare-reviewed papers taken up by pundits, democratic leaders, and the media as proof of what should happen in the future.
Unfortunately, according to research at — the irony is too much for fiction — the Federal Reserve, the results more often than not can’t be reproduced. One of the fundamental conditions of academic investigation is that others should be able to recreate one’s findings if they are usually accurate and not the result of some speculative accident. That basic appears to have left the area for a lengthy breakfast.
The report, which was first released through the Federal Reserve Board in 2015, should have had a more in-depth and perceptive resonance at the time. Even the name was terrible: Is Economics Research Replicable? Thirteen Journals’ published papers have “typically not” reported on sixty of them.
It was a team working paper. As a result, it was one of those “preliminary materials distributed to stimulate dialogue and important remark” that “does not indicate consent by other users of the research team or the Board of Governors.”
It is understandable that an organization based on the wisdom and perceptions of financial research does not agree with something that suggested, from a medical point of view, a lot of the work may be hooey.
The creators, Andrew Chang, now a principal analyst at the Board of Governors of the Federal Reserve System, and Phillip Li, a senior financial analyst at the Federal Deposit Insurance Corporation, tried to replicate 67 papers from 13 “well-regarded” economics journals.
First trouble, some of the journals required data and code and others didn’t. That left them obtaining data and code for 29 of 35 papers (83%) that required their supply, and 11 of 26 papers (42%), which did not. There were 6 documents that used personal data that was absent. (Adding the 35 required to provide data, 26 that weren’t, and 6 that used confidential data gets the total 67.)
Excluding the 6 documents with personal data and 2 more that used technology accessible to the in-authors, they were able to recreate 29 out of 59 papers, or 49%, with support from the authors.
We assert that finance research is typically not replicable because we are able to recreate less than half of the documents in our example even with the writers’ assistance, they wrote.
Given the number of name economists who repeatedly cited the Phillips Curve, a proven unreliable correlation, as the reason the nation needed higher unemployment to combat rising prices, it shouldn’t be surprising that there isn’t any concern over replicable research.
What surfaced this about nine-year-old paper was an article in The Economist about the rise and fall of “Freakonomics”, the book — and finally radio show, podcast, and what have you — that was supposed to highlight “the invisible side of everything.
In particular, the article made a touching claim in a touching chapter about how the legalization of abortion in 1973 caused crime to decline in the 1990s due to the large number of criminals who would follow after them.
“It was a classic of the clever-dick genre: an unflinching social scientist using data to come to a counterintuitive conclusion, and not shying away from offence,” The Economist wrote. “It was, however, wrong”. A coding error and the use of total arrests, rather than arrest rate to see the amount of crime per population, undermined the claims. According to other researchers, the decline started with women and not with men, and that no-fault divorce “may have played a bigger role.”
Most of the article is about the entangled issues of supporters and detractors and said that “subsequent papers often overturned results, even if, as in the case of those popularized in English spelling by Freakonomics, they had an afterlife as cocktail-party anecdotes.”
The article ended by saying that “the centuries-old economics questions are as fascinating as they always were.” The methods to investigate them are still a work in progress.
If the economy is making such a mess, perhaps there should be less obedience and more disregard for what they are pretty certain they know until they don’t.