the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.
Gelman & anonymous econ grad student
-
My problem with Gelman (as someone who uses nested models a lot) is that if you do what he proposes interpretation essentially becomes arbitrary. My second problem is that his textbook is straight up badly written. Third problem is he gets out over his skis a bit, especially with psychometrics where there are people who can and do challenge him, and he has trouble understanding the argument or at least answering the questions (see his total misunderstanding as support (?) of the existence of learning styles- which don’t exist)
-
I don’t think Gelman takes seriously the “100 page appendix and robustness check/sensitivity test” part. yes, you can p-hack that as well, but it quickly becomes hard to find results because you are exploring a huge space (searching for a result that is significant and comparable magnitude in 100 regressions). also, since many robustness checks are requested by reviewers, submitting a result you know to be spuriously robust is risky; they might ask for one of the robustness checks that will fail.
-
I don’t think Gelman takes seriously the “100 page appendix and robustness check/sensitivity test” part. yes, you can p-hack that as well, but it quickly becomes hard to find results because you are exploring a huge space (searching for a result that is significant and comparable magnitude in 100 regressions). also, since many robustness checks are requested by reviewers, submitting a result you know to be spuriously robust is risky; they might ask for one of the robustness checks that will fail.
lol, just fake the results
-stevie
-
P-hacking is bad but exploring the data is not. Say I get some new data and realize my initial research plan doesn't make sense because of important confounding factor. So instead I write a paper about the confounding factor or some unexpected relationship in the data that I discover is very robust. Is that poor research? Of course not.
-
There are journals experimenting with accepting papers based on a pre-analysis plan. The resulting papers are junk. It turns out people discover things when they explore the data in an open-ended way--who would have thought?
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY. -
There are journals experimenting with accepting papers based on a pre-analysis plan. The resulting papers are junk. It turns out people discover things when they explore the data in an open-ended way--who would have thought?
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.
"discover things" that are untrue. and this is what the profession wants. untrue results. who would have thought?
-
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.Sounds like you just admitted that empirical work is UNTRUE.
-
when you submit a paper to a pre-analysis plan can you do additional analysis outside the agreement?
say you ran an experiment with a pre-analysis plan and the promised t-test found no sig effect
but then you go on and discover that the treatment effect is in fact heterogenous in an unexpected yet illuminating way. could you publish that as well? would obviously make for a better paper
or is pre-analysis a binding contract from which you cannot step outside?
-
Say you're John Snow collecting data on malaria in the 1850s. You plan to test how miasma in the air contributes to malaria (which was the mainstream view in the day) but you find a strong correlation between sanitation and malaria outbreaks. Is this invalid?
There are journals experimenting with accepting papers based on a pre-analysis plan. The resulting papers are junk. It turns out people discover things when they explore the data in an open-ended way--who would have thought?
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.
"discover things" that are untrue. and this is what the profession wants. untrue results. who would have thought?
-
Say you're John Snow collecting data on malaria in the 1850s. You plan to test how miasma in the air contributes to malaria (which was the mainstream view in the day) but you find a strong correlation between sanitation and malaria outbreaks. Is this invalid?
First, it was cholera, not "malaria".
Second, Snow was first to reject the 'miasma' theory. He was an expert on respiration, and he reasoned that if the disease were due to miasma, then surely the lungs would be affect first. But holera is primarily a disease of the alimentary canal.
Third, that wasn't how he discovered the correlation between cholera and sanitation. It started with his statistical studies on the number of deaths in the various districts of London supplied by each of the different water companies. During the course of these studies, he focused on increasingly on the water supplies for specific geographic regions, until it became evident that people drinking from certain water pumps were getting the disease, while people drinking from other water pumps weren't. In his own words:
"The extraordinary eruption of cholera in the Soho district which was
carefully examined . . . does not appear to afford any exception to generalisations respecting local states of uncleanliness, overcrowding, and imperfect ventilation. The suddenness of the outbreak, the immediate climax and short duration, all point to some atmospheric or other widely
diffused agent still to be discovered, and forbid the assumption, inthis
instance, of any communication of the disease from person to person either by infection or contamination of water with the excretions of the sick."So, I have to ask, who the mind of a cIueless uneducated bIowhard like you functions?
It's as if you heard something about snow and cholera once in your life, and confabulated the details just now so you could assert your two-cents into a topic about which you clearly know nothing.
Why?
-
Say you're John Snow collecting data on malaria in the 1850s. You plan to test how miasma in the air contributes to malaria (which was the mainstream view in the day) but you find a strong correlation between sanitation and malaria outbreaks. Is this invalid?
There are journals experimenting with accepting papers based on a pre-analysis plan. The resulting papers are junk. It turns out people discover things when they explore the data in an open-ended way--who would have thought?
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.
"discover things" that are untrue. and this is what the profession wants. untrue results. who would have thought?
1e82 has already destroyed you. Nobody is preventing you from writing a "negative" paper showing that the proposed explanation does not stand the data AND to explicitly write another paper in which you try to "explore the data" and see if there is anything interesting AFTER you have specified ex-ante which regressions you are going to run and corrected for multiple testing.
-
The problem is that economists are on the d*mber side when it comes to science: taking a class or two in philosophy of science or in physics would help the profession tremendously.
Collecting facts and connections between facts is perfectly fine. Trying to explain a connection between facts by a very naive model is not. Trying to convince people that the connection you pointed out is an exclusive causal connection by cooking a nice story, backed by all the economic jargon and common sense derived from normative models is not.
If economists were only trying to garner connections between facts, everything would be good.
-
I don’t think Gelman takes seriously the “100 page appendix and robustness check/sensitivity test” part. yes, you can p-hack that as well, but it quickly becomes hard to find results because you are exploring a huge space (searching for a result that is significant and comparable magnitude in 100 regressions). also, since many robustness checks are requested by reviewers, submitting a result you know to be spuriously robust is risky; they might ask for one of the robustness checks that will fail.
If parameters are not precisely estimated (large standard errors) shouldn't you expect some robustness checks to be statistically insignificant?
-
So we agree that it's ok to explore the data (after writing up some negative results). Cool.
1e82 has already destroyed you. Nobody is preventing you from writing a "negative" paper showing that the proposed explanation does not stand the data AND to explicitly write another paper in which you try to "explore the data" and see if there is anything interesting AFTER you have specified ex-ante which regressions you are going to run and corrected for multiple testing.
-
Yes, he explored the data and found a correlation between water supplies in specific regions and disease.
It started with his statistical studies on the number of deaths in the various districts of London supplied by each of the different water companies. During the course of these studies, he focused on increasingly on the water supplies for specific geographic regions, until it became evident that people drinking from certain water pumps were getting the disease.
-
So statisticians built a tool that assumes that regressions are run once. Science doesn't work that way. So Gelman concludes that scientific practice is wrong.
There is nothing to learn from this thread, except that RA Fisher's tools are outdated, that 95% is arbitrary, and some insist that the entire profession is wrong because we can't depart from that framework.
Some people make a living out of this demagoguery.
-
In some case you might be right. But the alternative, when pre-analysis plans are not useful, is not to do the usual garden of forking paths, p-hack and contribute to publication bias. The alternative is e.g. some type of multiverse analysis. Read the articles in the links I posted above.
Don’t forget Gelmans following point:
”Take a perfectly fine, if noisy, experiment, run it through the statistical-significance filter (made worse by p-hacking, but often pretty bad even when only one analysis is done on your data), and you can end up with something close to a pile of random numbers. That’s not good for exploratory research either!”
At least I hope you agree that we do not learn much from a pile of random numbers?
the stu-pid thing is I doubt we’d learn more about the world if we were required first to state the exact specifications to run for the paper before ever accessing the data. In many to most cases it’d be gar-bage.
Empirical work is MESSY.