Paola is a coauthor of Yiming Cao, who is a grandstudent of James Kung. We are in a fraud pandemic!
Plus Falk and Enke. You can't make it up.
REStud retraction
-
Yes, the abstract does not reflect the replication results. Actual results show original paper results were an artifact.
The authors behind the replication are both German and one is male. See link to paper earlier in the thread. There's a lot more to the paper than the abstract says, the good stuff takes a couple of pages to get to, but it's also very short and concise.
-
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The point is not about replication. It is about pretending that all possible robustness tests, heterogeneity tests, tests on potential mechanisms, etc. work perfectly. In practice, what happens is that people show only the subset of results that looks fine in the best-case scenario, or fake results in the worst-case scenario. We should stop the "fine looking results" bias in publications. If the research question is good, the data is good, the method is good in principle but maybe it fails to be robust in that particular application, the paper should be accepted anyway by top journals.
-
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The Norwegian/Swedish admin data papers are a complete farce. If you want to use their data you are forced to team up with one of their dimwit PhD's/APs who's made a career out of massaging that particular admin data set.
-
This is true but this story is a completely different issue
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The point is not about replication. It is about pretending that all possible robustness tests, heterogeneity tests, tests on potential mechanisms, etc. work perfectly. In practice, what happens is that people show only the subset of results that looks fine in the best-case scenario, or fake results in the worst-case scenario. We should stop the "fine looking results" bias in publications. If the research question is good, the data is good, the method is good in principle but maybe it fails to be robust in that particular application, the paper should be accepted anyway by top journals.
-
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The point is not about replication. It is about pretending that all possible robustness tests, heterogeneity tests, tests on potential mechanisms, etc. work perfectly. In practice, what happens is that people show only the subset of results that looks fine in the best-case scenario, or fake results in the worst-case scenario. We should stop the "fine looking results" bias in publications. If the research question is good, the data is good, the method is good in principle but maybe it fails to be robust in that particular application, the paper should be accepted anyway by top journals.
It would indeed be much healthier if people felt comfortable being transparent. Yet for this to happen, editors need to signal that they would be okay to accept papers with ambiguous findings and speculative studies. Has any (good) journal proved willing to do so?
-
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The point is not about replication. It is about pretending that all possible robustness tests, heterogeneity tests, tests on potential mechanisms, etc. work perfectly. In practice, what happens is that people show only the subset of results that looks fine in the best-case scenario, or fake results in the worst-case scenario. We should stop the "fine looking results" bias in publications. If the research question is good, the data is good, the method is good in principle but maybe it fails to be robust in that particular application, the paper should be accepted anyway by top journals.
It would indeed be much healthier if people felt comfortable being transparent. Yet for this to happen, editors need to signal that they would be okay to accept papers with ambiguous findings and speculative studies. Has any (good) journal proved willing to do so?
only in very rare occasions (needless to say, option not available to anyone who isn't VHRM)
-
Imagine a country investing billions of its taxpayers money over several generations in administrative data of extremely high quality only to give it away for free to random foreign researchers. A country is supposed to serve its citizens first, and foreigners second. Not the other way around.I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The Norwegian/Swedish admin data papers are a complete farce. If you want to use their data you are forced to team up with one of their dimwit PhD's/APs who's made a career out of massaging that particular admin data set.
-
I'd like to look more generally at the problem, apart from this specific case. I think that empirical economics has a credibility problem because of editorial attitudes toward empirical results. I think that the credibility of empirical results would improve if there were top 5 papers with an abstract like the following (I've seen none so far):
We address this important and long-standing question in economics using this high quality dataset which is ideal to address this question. We adopt this estimation method which in principle is the most credible given the available data for the following reasons. Our main result is X. However, the overall solidity of this result is weak, since the parallel trend test /robustness test X fails.This is why we got an endless supply of Norwegian administrative data papers about 15 yrs ago. That just got boring, and was also non-replicable unless you had a Norwegian co-author. The best system would be high stakes for showing replication failure (top 5 with protection against retaliation, somehow) with a ban on "secret" data. Every time I want to run replication code, I run into "get permission from agency x to obtain the data ...." Replication should be as easy as possible to make the cheating certainly discoverable.
The Norwegian/Swedish admin data papers are a complete farce. If you want to use their data you are forced to team up with one of their dimwit PhD's/APs who's made a career out of massaging that particular admin data set.
Imagine a country investing billions of its taxpayers money over several generations in administrative data of extremely high quality only to give it away for free to random foreign researchers. A country is supposed to serve its citizens first, and foreigners second. Not the other way around.
'extremely high quality' . Right mogstad.