''Yes, it’s fine to chase noise, but then you should chase noise in all directions, i.e., do a multiverse analysis. The mistake is when a researcher picks out one particular piece of noise and grabs on to it, while ignoring or downplaying all the other correlations in the data.''
Love this comment by Andrew below this post:
Theorists >>> reg monkeysAnd yet, reg monkeys are the majority in most departments, and their cutesy studies dominate the public discourse.
You say “and yet” but actually the fact theory >>> ref monkeys IMPLIES that there will be less theorists and more ref monkeys
I also wish nonparametric methods were used more often, as a way to decrease the degrees of freedom resulting from making assumptions about functional form and (when these methods allow for it) variable selection. This is an area where using ML to aid causal inference is not a meme but a potentially useful thing, besides other more traditional uses (e.g. imputation of missing data, another task that may actually be important). It could also be used to justify using a specific functional form and using a parametric method to estimate it, if the results warranted doing so.
This doesn't mean multiverse analysis would not be useful, particularly when it comes to data cleaning. And it would also not be feasible if the samples are not large enough, since nonparametric methods tend to be data hungry given their slower convergence rates. A multiverse analysis might as well take longer since they also tend to be more computationally demanding.
And yet, I can't help but feel nonparametric methods may still be useful to alleviate the issue of the unspecified researchers degrees of freedom.Interesting. Could you perhaps elaborate? Can your recommend any literature?
Sorry, I just saw your response. Just read the Double-ML literature.
Here, you often assume a binary treatment and the ML methods - which are often nonparametric, like all tree-based methods and neural networks - can take care of the selection of other controls and also of their functional form.
The binary treatment assumption is not binding, but it illustrates how they can be useful in this regard.
Cheers.
I also wish nonparametric methods were used more often, as a way to decrease the degrees of freedom resulting from making assumptions about functional form and (when these methods allow for it) variable selection. This is an area where using ML to aid causal inference is not a meme but a potentially useful thing, besides other more traditional uses (e.g. imputation of missing data, another task that may actually be important). It could also be used to justify using a specific functional form and using a parametric method to estimate it, if the results warranted doing so.
This doesn't mean multiverse analysis would not be useful, particularly when it comes to data cleaning. And it would also not be feasible if the samples are not large enough, since nonparametric methods tend to be data hungry given their slower convergence rates. A multiverse analysis might as well take longer since they also tend to be more computationally demanding.
And yet, I can't help but feel nonparametric methods may still be useful to alleviate the issue of the unspecified researchers degrees of freedom.Interesting. Could you perhaps elaborate? Can your recommend any literature?
Sorry, I just saw your response. Just read the Double-ML literature.
Here, you often assume a binary treatment and the ML methods - which are often nonparametric, like all tree-based methods and neural networks - can take care of the selection of other controls and also of their functional form.
The binary treatment assumption is not binding, but it illustrates how they can be useful in this regard.
Never forget the discussion between Andrew Gelman and Richard Tol. That’s when my trust in academia finally died. It’s completely fascinating how Richard Tol could just get away with that stuff. None of his coworkers seemed to care one bit about the way that he embarrassed himself. And then Nordhaus just made the whole story even worse. How are we supposed to trust any of them?
Lol. Wrong how? Have you studied any econ beyond undergraduate intro stuff?
Or, say, if I want to estimate the effect of a raise in the minimum wage on employment. Mainstream econ models may predict that it causes increased employment, reduced employment, or neither. Right?Wrong. I hope you are not an economist.
Most people in here lack even the most basic understanding of statistics. All these comments reassuring us that everything’s fine, as long as your RA managed to produce a ''convincing'' battery of ''robustness checks'' for the online appendix by essentially solving a sudoku (as one user quite brilliantly described it). Of course, after already spending hundreds of hours in the garden of forking paths chasing particular pieces of noise.
And then they wonder why nobody trusts social ”scientists”.
That’d leave us with approx. 15 people. The upside is that everyone can have tenure and publish in QJE.
We need more posts like this, in the media and elsewhere. Ideally including the full names of each cheating researcher and professor. They need to be removed from academia.
You should read the threads in this post before commenting. For example, the garden of forking paths is not the same as p-hacking.
If you think p hacking is cheating, oh boy are you also a cheater
We need more posts like this, in the media and elsewhere. Ideally including the full names of each cheating researcher and professor. They need to be removed from academia.
3rd year PhD student here. One of the undergrad RAs employed by my supervisor has repeatedly made me aware of peculiar errors in raw data files and code compiled by my supervisor when putting together a massive dataset. This dataset is central to a top 5 submission (current status R&R) co-authored by my supervisor, a last-year PhD student and 2 HRMs. I myself co-author a paper with my supervisor using variables from the dataset for central analyses. I couldn’t let it go and just had to take a deeper look into some of the underlying files and code. First I didn’t want to believe it, but the truth is that thanks to this RA I ended up discovering indisputable evidence of research fraud. Examples include replacing scans of authentic historical documents with fabricated ones before extracting/converting to text/data, generating 500 completely made up observations to make multiple main results stronger, using a figure in the appendix that relies on a fake regressor variable generated by a few lines of code in a dofile, and excluding multiple estimates that do not support the main conclusions of the paper.
What should I do? I honestly don’t know what to do. Who do I talk to?