Adler v. Chetty/Hoynes

How is it reasonable when the coauthor (Rockoff) admitted to result misrepresentation verbally but the issue was never addressed?
Adler's response to the response:
http://nepc.colorado.edu/thinktank/reviewmeasuringimpactofteachers"Chetty et al. inflate their results by a factor of 10 throughout their paper. Before I
published my review of this paper I wrote to Jonah Rockoff about this problem as it
applies to test scores. His written response is quoted and discussed below, where I
examine the inflation of the effect of teacher valueadded on test scores.2 The fact that they
repeat this misrepresentation after it was called to their attention is troubling."The footnote points to another document: http://epaa.asu.edu/ojs/article/view/1264/1033
" While this may have been an honest
mistake, when asked about this, one of the authors, Jonah Rockoff, responded: “This is definitely a
language error on our part. Instead of 'a one SD increase in teacher quality' we should have said 'an
increase in teacher valueadded of one (studentlevel) standard deviation.' Of course an increase of
one in value added is roughly 10 teacherlevel standard deviations, so your assessment of 0.03 in year
three for a one teacherlevel SD increase in year 0 would be correct'" (Jonah Rockoff to Moshe
Adler, personal communication, October 12, 2012). This answer is problematic, however."All seem to be shown here http://www.columbia.edu/~ma820/

Oh my god, Raj write emails like this?
Dear Professor Adler,
I thought you might be interested in the final versions of our teacher
valueadded papers, which are now forthcoming in the American Economic
Review:http://obs.rc.fas.harvard.edu/chetty/w19423.pdf
http://obs.rc.fas.harvard.edu/chetty/w19424.pdfThese papers were previously combined as a single paper (NBER wp
17699). When we submitted the paper to the AER, the editor and all
three referees said they felt there were two AER quality articles here
and encouraged us to develop them separately for publication
backtoback in the AER. To our knowledge, this is one of the first
times this has occurred in economics since Peter Diamond and Jim
Mirrlees published their seminal work on optimal taxation in 1971,
which later led to two Nobel prizes.You might also be interested in the recent papers by Kane, Staiger,
and BacherHicks, who replicate the results of our first paper in the
LA school district, and Chamberlain (PNAS 2013), who replicated the
results on longterm impacts for a paper he wrote when he was elected
to the National Academy of Sciences.Best regards,
Raj Chetty

Oh my god, Raj write emails like this?
Dear Professor Adler,
I thought you might be interested in the final versions of our teacher
valueadded papers, which are now forthcoming in the American Economic
Review:
http://obs.rc.fas.harvard.edu/chetty/w19423.pdf
http://obs.rc.fas.harvard.edu/chetty/w19424.pdf
These papers were previously combined as a single paper (NBER wp
17699). When we submitted the paper to the AER, the editor and all
three referees said they felt there were two AER quality articles here
and encouraged us to develop them separately for publication
backtoback in the AER. To our knowledge, this is one of the first
times this has occurred in economics since Peter Diamond and Jim
Mirrlees published their seminal work on optimal taxation in 1971,
which later led to two Nobel prizes.
You might also be interested in the recent papers by Kane, Staiger,
and BacherHicks, who replicate the results of our first paper in the
LA school district, and Chamberlain (PNAS 2013), who replicated the
results on longterm impacts for a paper he wrote when he was elected
to the National Academy of Sciences.
Best regards,
Raj ChettyAh0le factor: High

He compared his stuff to a paper that generated two nobel prizes. Talk about letting his ego go to his head!
Oh my god, Raj write emails like this?
Dear Professor Adler,
I thought you might be interested in the final versions of our teacher
valueadded papers, which are now forthcoming in the American Economic
Review:
http://obs.rc.fas.harvard.edu/chetty/w19423.pdf
http://obs.rc.fas.harvard.edu/chetty/w19424.pdf
These papers were previously combined as a single paper (NBER wp
17699). When we submitted the paper to the AER, the editor and all
three referees said they felt there were two AER quality articles here
and encouraged us to develop them separately for publication
backtoback in the AER. To our knowledge, this is one of the first
times this has occurred in economics since Peter Diamond and Jim
Mirrlees published their seminal work on optimal taxation in 1971,
which later led to two Nobel prizes.
You might also be interested in the recent papers by Kane, Staiger,
and BacherHicks, who replicate the results of our first paper in the
LA school district, and Chamberlain (PNAS 2013), who replicated the
results on longterm impacts for a paper he wrote when he was elected
to the National Academy of Sciences.
Best regards,
Raj Chetty 
Adler's points are reasonable. Seems valid. His critique in slideshow from his webpage http://www.columbia.edu/~ma820/Chetty,%20Kane%20Vergara.pptx
How is this any better than "ruptures" when all Hoynes did was to
forward the critiques from Adler to the authors, when they were clearly already in touch?And what is Chetty's email supposed to mean? He's right because his paper (or he himself?) is like Diamond and Mirrlees?

In the original version of our paper (NBER wp 17699, Table 6, Column 2), we did report estimates at age 30. The estimated impact at age 30 is $2,058, larger than the estimated impact at age 28 ($1,815). Why does Adler conclude that there is “no impact” at age 30 when the impact is actually larger? The reason is that
Adler confuses statistical significance with magnitudes: the standard error of the estimate at age 30 is $1,953, and hence is statistically insignificant (i.e., one cannot reject the hypothesis that the effect is zero). However,
this does not mean that there is no effect at age 30; rather, it means that one has insufficient data to measure earnings impacts accurately at age 30. Adler claims that the sample is “adequately large” based on a power calculation that assumes independent errors across observations and hence greatly overstates precision by failing to account for correlated errors across students within a classroom and repeat observations for students over time. The standard errors we estimate account for these issues and are a direct estimate of precision at
age 30 in the data; indeed, the difference in the standard errors between age 28 and 30 is exactly what one would expect given the reduction in sample size. In the revised version of the paper, we dropped the estimates at age 30 in the interest of space since there is inadequate data at age 30 to obtain precise estimates.I facepalmed and stopped reading after this.
Adler: You found no effect at age 30.
CFR: Nah, bro, we found a YUGE effect but standard errors, ya dig? 
What an arrogance. The AER rejects 95% of submissions from the whole world and the editor asks to divide the paper in two, both published in AER.
The author is not happy enough for two AERs and compares this to Diamond and Mirlees and their Nobel prizes!Isn't this really too much?

I facepalmed and stopped reading after this.
Adler: You found no effect at age 30.
CFR: Nah, bro, we found a YUGE effect but standard errors, ya dig?Read the response to the response as well http://nepc.colorado.edu/files/nepcttrfoloadlerresponse.pdf. Both sides are quoted in here.
Plus magnitudes are exaggerated by 10: " Of course an increase of one in value added is roughly 10 teacherlevel standard deviations, so your assessment of 0.03 in year three for a one teacherlevel SD increase in year 0 would be correct'" (Jonah Rockoff to Moshe Adler, personal communication, October 12, 2012)."
Adler does the best he can without having the data himself.

In the original version of our paper (NBER wp 17699, Table 6, Column 2), we did report estimates at age 30. The estimated impact at age 30 is $2,058, larger than the estimated impact at age 28 ($1,815). Why does Adler conclude that there is “no impact” at age 30 when the impact is actually larger? The reason is that
Adler confuses statistical significance with magnitudes: the standard error of the estimate at age 30 is $1,953, and hence is statistically insignificant (i.e., one cannot reject the hypothesis that the effect is zero). However,
this does not mean that there is no effect at age 30; rather, it means that one has insufficient data to measure earnings impacts accurately at age 30. Adler claims that the sample is “adequately large” based on a power calculation that assumes independent errors across observations and hence greatly overstates precision by failing to account for correlated errors across students within a classroom and repeat observations for students over time. The standard errors we estimate account for these issues and are a direct estimate of precision at
age 30 in the data; indeed, the difference in the standard errors between age 28 and 30 is exactly what one would expect given the reduction in sample size. In the revised version of the paper, we dropped the estimates at age 30 in the interest of space since there is inadequate data at age 30 to obtain precise estimates.I facepalmed and stopped reading after this.
Adler: You found no effect at age 30.
CFR: Nah, bro, we found a YUGE effect but standard errors, ya dig?
</blockquote
That's the weakest part of Adler's critique. If the confidence interval for 30yearolds is big it just means they can't say much about them with the data they have, not that the estimate for 28yearolds is wrong or even different.By Adler's logic, adding 50 observations at another age should invalidate the study almost no matter what the estimate for that group.