Close

Complete Captcha

• 1 year - 110 posts - Latest - RSS
• Thread: 284 Goods vs 113 No Goods

1. Economist
   c273
   

   The paper titled "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results" is currently the most cited 2019 article in the Quarterly Journal of Economics (372 Google cites).

   It delivers bad news to economists running experiments: their p-values are wrong. To get correct p-values, the article explains, they need to run, instead of regressions, something called "randomization tests".

   In this post I show that this conclusion only holds when relying on an unfortunate default setting in STATA:

   http://datacolada.org/99

   1 year ago # QUOTE 30 Volod 0 Vlad !

2. Economist
   3f5b
   

   Big if true

   1 year ago # QUOTE 4 Volod 0 Vlad !

3. Economist
   ee11
   

   Great article, and definitely big if true as other poster said. Sometimes I worry about how behind our stats is even though we seem to think we are trained appropriately.

   1 year ago # QUOTE 0 Volod 0 Vlad !

4. Economist
   6e32
   

   Great datacolada post. STATA is dangerous!

   1 year ago # QUOTE 3 Volod 2 Vlad !

5. Economist
   def7
   

   "It turns out that there are five main ways to compute robust standard errors. STATA has a default way, and it is not the best way.

   In other words, when in STATA you run: reg y x, robust you are not actually getting the most robust results available. Instead, you are doing something we have known to be not quite good enough since the first George W Bosh administration (the in-hindsight good old days).

   The five approaches for computing robust standard errors are unhelpfully referred to as HC0, HC1, HC2, HC3, and HC4. STATA's default is HC1. That is the procedure the QJE article used to conclude regression results are inferior to randomization tests, but HC1 is known to perform poorly with 'small' samples. Long & Ervin 2000 unambiguously write "When N<250 . . . HC3 should be used". A third of the simulations in the QJE paper have N=20, another third N=200."

   Economists are just second-raters all around.

   Theorists are second-rate, failed mathematicians.
   Econometricians are second-rate, failed statisticians.
   Even reg monkeys are second-rate, failed data scientists.

   1 year ago # QUOTE 16 Volod 8 Vlad !

6. Economist
   b902
   

   "It turns out that there are five main ways to compute robust standard errors. STATA has a default way, and it is not the best way.
   In other words, when in STATA you run: reg y x, robust you are not actually getting the most robust results available. Instead, you are doing something we have known to be not quite good enough since the first George W Bosh administration (the in-hindsight good old days).
   The five approaches for computing robust standard errors are unhelpfully referred to as HC0, HC1, HC2, HC3, and HC4. STATA's default is HC1. That is the procedure the QJE article used to conclude regression results are inferior to randomization tests, but HC1 is known to perform poorly with 'small' samples. Long & Ervin 2000 unambiguously write "When N<250 . . . HC3 should be used". A third of the simulations in the QJE paper have N=20, another third N=200."
   Economists are just second-raters all around.
   Theorists are second-rate, failed mathematicians.
   Econometricians are second-rate, failed statisticians.
   Even reg monkeys are second-rate, failed data scientists.

   "It turns out that there are five main ways to compute robust standard errors. STATA has a default way, and it is not the best way.
   In other words, when in STATA you run: reg y x, robust you are not actually getting the most robust results available. Instead, you are doing something we have known to be not quite good enough since the first George W Bosh administration (the in-hindsight good old days).
   The five approaches for computing robust standard errors are unhelpfully referred to as HC0, HC1, HC2, HC3, and HC4. STATA's default is HC1. That is the procedure the QJE article used to conclude regression results are inferior to randomization tests, but HC1 is known to perform poorly with 'small' samples. Long & Ervin 2000 unambiguously write "When N<250 . . . HC3 should be used". A third of the simulations in the QJE paper have N=20, another third N=200."
   Economists are just second-raters all around.
   Theorists are second-rate, failed mathematicians.
   Econometricians are second-rate, failed statisticians.
   Even reg monkeys are second-rate, failed data scientists.

   This would deserve to be pinned. I bet almost no one is aware of this inadequate default. I surely was not

   1 year ago # QUOTE 4 Volod 2 Vlad !

7. Economist
   5673
   

   This would deserve to be pinned. I bet almost no one is aware of this inadequate default. I surely was not

   Same. Had no clue.

   1 year ago # QUOTE 0 Volod 0 Vlad !

8. Economist
   f082
   

   These are the things I wish they taught me in grad school econometrics instead of theoretical time series

   1 year ago # QUOTE 13 Volod 0 Vlad !

9. Economist
   e1ea
   

   Shows you why regmonkeys have no business making bold claims about metrics. And why qje is a terrible journal that should be ignored

   1 year ago # QUOTE 7 Volod 4 Vlad !

10. Economist
    53e0
    

    Does it mean that if I rerun my stuff it will become significant?

    1 year ago # QUOTE 6 Volod 0 Vlad !

11. Economist
    89a9
    

    I know R custered standard error uses HC3 which is different to Stata default.

    But they dont differ much?

    1 year ago # QUOTE 1 Volod 0 Vlad !

12. Economist
    bd94
    

    losing hoap: reg y x, robust
    gaining hoap: reg y x, vce(hc3)

    1 year ago # QUOTE 15 Volod 1 Vlad !

13. Economist
    e47a
    

    Oh man, this is almost as bad as when all those early fMRI results were found to be overstated because SPM, the software everybody used, had random effects as the default setting and cognitive scientists had no experience with panel data.

    1 year ago # QUOTE 2 Volod 0 Vlad !

14. Economist
    82dd
    

    You frequentists are weird

    1 year ago # QUOTE 3 Volod 4 Vlad !

15. Economist
    3f5b
    

    "It turns out that there are five main ways to compute robust standard errors. STATA has a default way, and it is not the best way.
    In other words, when in STATA you run: reg y x, robust you are not actually getting the most robust results available. Instead, you are doing something we have known to be not quite good enough since the first George W Bosh administration (the in-hindsight good old days).
    The five approaches for computing robust standard errors are unhelpfully referred to as HC0, HC1, HC2, HC3, and HC4. STATA's default is HC1. That is the procedure the QJE article used to conclude regression results are inferior to randomization tests, but HC1 is known to perform poorly with 'small' samples. Long & Ervin 2000 unambiguously write "When N<250 . . . HC3 should be used". A third of the simulations in the QJE paper have N=20, another third N=200."
    Economists are just second-raters all around.
    Theorists are second-rate, failed mathematicians.
    Econometricians are second-rate, failed statisticians.
    Even reg monkeys are second-rate, failed data scientists.

    This would deserve to be pinned. I bet almost no one is aware of this inadequate default. I surely was not

    HC3 literally was published with the original HC1 that is Stata's default.

    https://doi.org/10.1016/0304-4076(85)90158-7

    The problem, indeed, is reg monkeys not being aware of what , robust actually does.

    1 year ago # QUOTE 5 Volod 0 Vlad !

16. Economist
    6e17
    

    but, but my "identification strategy" gives me good results with robust....and no stars with HC3.

    1 year ago # QUOTE 10 Volod 0 Vlad !

17. Economist
    6d38
    

    Few know this

    1 year ago # QUOTE 0 Volod 0 Vlad !

18. Economist
    714d
    

    "HC3. Is it possible to learn this power?"

    "Not from an economist."

    1 year ago # QUOTE 5 Volod 0 Vlad !

19. Economist
    770f
    

    I mean, there's a bit of a "this statement is not true" thing here - if most economists are making the same mistake AY is making by using the default standard errors, then AY's analysis is mostly correct.

    1 year ago # QUOTE 7 Volod 0 Vlad !

20. Economist
    2bc5
    

    Also, I would bet that the major inference problems in RCTs come from clustering, not heteroscedasticity. Specifically, using vce(cluster whatever) when the number of clusters is not big enough to justify the asymptotics.

    1 year ago # QUOTE 8 Volod 1 Vlad !

Reply »