Strategically omitted references in Spiess (2017) JMP

3 years - 325 posts - Latest - RSS
Thread: 2389 Goods vs 1901 No Goods

Economics Job Market Rumors » Economics » Economics Discussion

Strategically omitted references in Spiess (2017) JMP

Economist
31dd

What's the earliest version of the Wu and Gagnon-Bartsch paper? Arxv shows August 2017 as the most recent version.

WBG is a working paper that was posted in arvix on August 3, 2017. It's perfectly possible for Spiess to be unaware of it until November or December.

By the way, WBG doesn't cite Spiess either at any point in the text. And the paper lacks a list of references at the end.

OP links to the draft one of the authors posted on his own website, December 2016 http://www-personal.umich.edu/~johanngb/working/loop.pdf

This and the arxiv draft both DO have a list of references, after the main text, before the appendix. They could not have cited Spiess since his paper did not exist even when they posted on arxiv.

3 years ago # QUOTE 48 Jab 0 No Jab !
Economist
9e08

Just wondering which one of the top departments will give JS an offer.

Spiess is a joke, a complete joke. He is not a researcher, more like a trump type politician

3 years ago # QUOTE 0 Jab 0 No Jab !
Economist
e2f3

He's a scam, but will still place better than any of you guys, so get a life

3 years ago # QUOTE 2 Jab 17 No Jab !
Economist
56a9

Leave the poor guy alone. It's the opposite of what you're claiming. The other papers copied him, not the other way round.
OP, get a life. Instead of nitpicking other people's papers with jealousy, focus on your own papers. You wouldn't have failed the market if you focused on yourself.

December 3, 2016
http://www-personal.umich.edu/~johanngb/working/loop.pdf

If these two copied Spiess, then provide a copy of Spiess's paper that predates December 3, 2016.

3 years ago # QUOTE 22 Jab 0 No Jab !
Economist
d33f

Leave the poor guy alone. It's the opposite of what you're claiming. The other papers copied him, not the other way round.
OP, get a life. Instead of nitpicking other people's papers with jealousy, focus on your own papers. You wouldn't have failed the market if you focused on yourself.

December 3, 2016
http://www-personal.umich.edu/~johanngb/working/loop.pdf
If these two copied Spiess, then provide a copy of Spiess's paper that predates December 3, 2016.

It was circulated to those authors when it was in a preliminary draft version. If you don't know the facts STFU.

3 years ago # QUOTE 3 Jab 39 No Jab !
Economist
0d3a

Lol I knew speiss was a fraud. I'm at one of the schools that he had a flyout at and I'm tempted to forward this to the hiring committee.

3 years ago # QUOTE 45 Jab 1 No Jab !
Economist
c0f6

didnt read a single line but thanks OP, thats what ejmr should be mostly about.

3 years ago # QUOTE 14 Jab 0 No Jab !
Economist
56a9

Leave the poor guy alone. It's the opposite of what you're claiming. The other papers copied him, not the other way round.
OP, get a life. Instead of nitpicking other people's papers with jealousy, focus on your own papers. You wouldn't have failed the market if you focused on yourself.

December 3, 2016
http://www-personal.umich.edu/~johanngb/working/loop.pdf
If these two copied Spiess, then provide a copy of Spiess's paper that predates December 3, 2016.

It was circulated to those authors when it was in a preliminary draft version. If you don't know the facts STFU.

Upload a copy of that preliminary draft version.

3 years ago # QUOTE 3 Jab 0 No Jab !
Economist
407b

Yeah let's criticize someone for doing the right thing and cite prominently a competing paper.

3 years ago # QUOTE 3 Jab 31 No Jab !
Economist
2e00

hiring committee holding special meeting to determine if his offer will be revoked

3 years ago # QUOTE 1 Jab 1 No Jab !
Economist
56a9

Yeah let's criticize someone for doing the right thing and cite prominently a competing paper.

It looks like plagiarism.

3 years ago # QUOTE 15 Jab 1 No Jab !
Economist
407b

hiring committee holding special meeting to determine if his offer will be revoked

GTFO undergrad

3 years ago # QUOTE 2 Jab 0 No Jab !
Economist
9ca1

I don't want to defend Spiess too hard since this does look troubling, but it is possible that in working on the same broad question the two groups of authors found similar approaches to the topic since such things are sometimes obvious.

The lack of citation is most definitely a cause for concern. On the other hand, the stuff which (so far) has been quoted as comparisons between the two papers seems (to me) mostly to be in the category of "the most obvious way to handle a pretty basic issue."

For a comparison: two IO guys working independently who wanted to learn something about demand might well start by estimating very similar discrete choice models. Two regmonkeys might stumble upon the same IV - and using *some* IV is definitely going to be their first idea.

If later in the paper there is a genuinely non-obvious idea from WGB (granted: not easy to judge or identify and will vary by reader!) which Spiess could only have lifted, that would be a very different story.

3 years ago # QUOTE 17 Jab 4 No Jab !
Economist
99a0

hiring committee holding special meeting to determine if his offer will be revoked

On a Sunday? 4 hours after OP post? Get a hold of yourself.

3 years ago # QUOTE 2 Jab 0 No Jab !
Economist
4dd0

hiring committee holding special meeting to determine if his offer will be revoked

On a Sunday? 4 hours after OP post? Get a hold of yourself.

what offers does he have

3 years ago # QUOTE 0 Jab 0 No Jab !
Economist
c67c

Here is a line by line takedown of your libel.

I was reading the intro to Spiess's JMP (December 8, 2017 version) and noticed in footnote 5 a reference to the "leave-one-out potential outcomes" (LOOP) estimator from Wu and Gagnon-Bartsch (WGB). I was curious to learn more since I am not familiar with these topics and noticed the title of the paper "The LOOP Estimator: Adjusting for Covariates in Randomized Experiments" which sounded highly relevant for Spiess but was only cited in footnotes and was not mentioned anywhere in the literature review section.

Simple explanation: He didn't know about the paper before and now he knows and cites properly.

I looked through WGB [http://www-personal.umich.edu/~johanngb/working/loop.pdf] and in particular read through section 3.
When I turned back to Spiess's section 1 ("A Simple Example"), I noticed that what I was reading sounded very familiar.
First is notation (WGB --> Spiess):
i's treatment assignment: X_i --> d_i
assume treatment is assigned randomly: p_i = P(X_i = 1) --> p = P(d_i = 1)
potential outcome if i is assigned to treatment: t_i --> y_i(1)
potential outcome if i is assigned to control: c_i --> y_i(0)
individual treatment effect / unit-level causal effect: tau_i --> tau_i

OP, why did you feel the need to write this post if you aren't familiar with the treatment effects literature? This notation and framework is used by almost everyone in this field just as people use Y = alpha + beta X + epsilon for OLS. I hope you don't claim that 95% of the papers that use beta for OLS coefficient plagiarized.

3 years ago # QUOTE 27 Jab 41 No Jab !
Economist
44e2

Who cares... None of this stuff is either interesting or important anyway

I got a copy of the JMP by Jann Spiess in December to look at sometime. I had to look at multiple versions because it kept on being updated, even until the end of flyout season in February. Here are some of the versions.
November 29, 2017 version: https://www.gsb.stanford.edu/sites/gsb/files/jmp_jann-spiess.pdf
December 8, 2017 version: https://economics.yale.edu/sites/default/files/jmp_3.pdf
February 4, 2018 version: https://scholar.harvard.edu/files/spiess/files/spiess_jmp_feb4.pdf
-----
I was reading the intro to Spiess's JMP (December 8, 2017 version) and noticed in footnote 5 a reference to the "leave-one-out potential outcomes" (LOOP) estimator from Wu and Gagnon-Bartsch (WGB). I was curious to learn more since I am not familiar with these topics and noticed the title of the paper "The LOOP Estimator: Adjusting for Covariates in Randomized Experiments" which sounded highly relevant for Spiess but was only cited in footnotes and was not mentioned anywhere in the literature review section.
I looked through WGB [http://www-personal.umich.edu/~johanngb/working/loop.pdf] and in particular read through section 3.
When I turned back to Spiess's section 1 ("A Simple Example"), I noticed that what I was reading sounded very familiar.
First is notation (WGB --> Spiess):
i's treatment assignment: X_i --> d_i
assume treatment is assigned randomly: p_i = P(X_i = 1) --> p = P(d_i = 1)
potential outcome if i is assigned to treatment: t_i --> y_i(1)
potential outcome if i is assigned to control: c_i --> y_i(0)
individual treatment effect / unit-level causal effect: tau_i --> tau_i
Next is the observation that it is "not possible to observe any single participant's treatment effect, because for each participant we are only able to observe the treatment response or the control response" (WGB) compared to "We do not observe both potential outcomes for one unit simultaneously, but observe only the treatment status and the realized outcome" (Spiess).
Next, "Less well known is the fact that it is also possible to provide an unbiased estimate of an individual participant's treatment effect. For example, Y_i*U_i is one such estimator" (WGB) compared to "we can still obtain an unbiased estimate of the unit treatment effect. Indeed, I will argue below that (d_i-p)/(p(1-p))y is an unbiased estimator for tau_i" (Spiess). Compare the equation at the top of Section 1.3 in Spiess to Equation (9) in WGB. The line following that equation in Spiess shows why it's unbiased, identical to Equation (10) and Equation (11) in WGB.
Next, "Although this does result in an unbiased estimator of tau_i, it is clearly useless for all practical purposes. A more sanguine way of putting this would be that the estimator, despite being unbiased, likely has very high variance" (WGB) compared to "But this estimator can have very high variance" (Spiess).
Next, "As an alternative estimator of tau_i, consider [Equation (12)]" (WGB) compared to "We can modify this estimator by regression adjustments y-hat to obtain [Equation (1)]" (Spiess). The equations are again the same, with the adjustment called m-hat in WGB and y-hat in Spiess.
Next, "if m-hat is independent of the i-th participant's treatment assignment - then tau_i-hat is an unbiased estimator of tau" (WGB) compared to "As long as y_i-hat only uses information from x_i and z_-i and not the outcome y_i or treatment assignment d_i, tau_i-hat will still be unbiased" (Spiess).
Next, "The advantage of this estimator is that it wil...See full post

3 years ago # QUOTE 0 Jab 17 No Jab !
Economist
c67c

Next is the observation that it is "not possible to observe any single participant's treatment effect, because for each participant we are only able to observe the treatment response or the control response" (WGB) compared to "We do not observe both potential outcomes for one unit simultaneously, but observe only the treatment status and the realized outcome" (Spiess).

This is Treatment Effects (TE) 101. Look up any TE paper and you'll find a variation of the exact same sentence.

Next, "Less well known is the fact that it is also possible to provide an unbiased estimate of an individual participant's treatment effect. For example, Y_i*U_i is one such estimator" (WGB) compared to "we can still obtain an unbiased estimate of the unit treatment effect. Indeed, I will argue below that (d_i-p)/(p(1-p))y is an unbiased estimator for tau_i" (Spiess). Compare the equation at the top of Section 1.3 in Spiess to Equation (9) in WGB. The line following that equation in Spiess shows why it's unbiased, identical to Equation (10) and Equation (11) in WGB.

It's called a propensity score weighted correction, and again this is a widely stated result in TE papers, usually without citation because it's so well known and widely used.

3 years ago # QUOTE 20 Jab 41 No Jab !
Economist
c67c

Next, "Although this does result in an unbiased estimator of tau_i, it is clearly useless for all practical purposes. A more sanguine way of putting this would be that the estimator, despite being unbiased, likely has very high variance" (WGB) compared to "But this estimator can have very high variance" (Spiess).
Next, "As an alternative estimator of tau_i, consider [Equation (12)]" (WGB) compared to "We can modify this estimator by regression adjustments y-hat to obtain [Equation (1)]" (Spiess). The equations are again the same, with the adjustment called m-hat in WGB and y-hat in Spiess.

I hate that I have repeat it, but again this is standard notation, at least in the statistics literature dealing with TE and talking about regression adjusted treatment effects.

Next, "if m-hat is independent of the i-th participant's treatment assignment - then tau_i-hat is an unbiased estimator of tau" (WGB) compared to "As long as y_i-hat only uses information from x_i and z_-i and not the outcome y_i or treatment assignment d_i, tau_i-hat will still be unbiased" (Spiess).
Next, "The advantage of this estimator is that it will have a low variance as long as m-hat ~= m" (WGB) compared to "Which regression adjustment minimizes variance? Optimally the investigator would set y-hat to (1-p)y_i(1)+py_i(0)" (Spiess). Equation (6) in WGB tells us that m is the same as the object that Spiess is setting y-hat to.

What is wrong with you? The notation is standard in the literature. If you don't understand the new theorems the papers prove, then GTFO. Don't come here complaining without understanding the results. If you have proof that his theorems were copied, point it out here and call him out. But this forum enables baseless libel like yours.

3 years ago # QUOTE 14 Jab 43 No Jab !
Economist
c67c

I don't know about the rest, but besides the similarity in the exposition, another peculiar issue is how the references to WGB have changed over time.
November 29, 2017 version of Spiess does not cite WGB at all. This is in spite of the fact that the associated content (for example, what is quoted above from section 1) has always been the same.
December 8, 2017 version of Spiess adds WGB to reference list and mentions the paper in 4 footnotes.
January 28, 2018 version of Spiess has just the same footnote mentions of WGB as earlier.
February 4, 2018 version of Spiess adds references to WGB throughout the main text, now mentioned 14 times in the paper. Here are some examples of how things have changed:
- New version adds a new paragraph in the literature review containing "Most closely related to the investigator's solution in my paper, Wu and Gagnon-Bartsch (2017)..."
- January 28: "The proposed mechanism therefore allows researchers to leverage machine learning in estimating average treatment effects in experimental data."
- February 4: "As in Wager et al. (2016) and Wu and Gagnon-Bartsch (2017), the sample-splitting construction allows researchers to leverage machine learning in estimating average treatment effects in experimental data."
- January 28: "As long as y_i-hat only uses information from x_i and z_-i and not the outcome y_i or treatment assignment d_i, tau_i-hat will still be unbiased."
- February 4: "As long as y_i-hat only uses information from x_i and z_-i and not the outcome y_i or treatment assignment d_i, tau_i-hat will still be unbiased... Wu and Gagnon-Bartsch (2017) introduce this estimator as the "leave-one out potential outcomes" (LOOP) estimator."
- January 28: "Which regression adjustment minimizes variance? Optimally the investigator would set..."
- February 4: "Among unbiased estimators, which regression adjustment minimizes variance? As Wu and Gagnon-Bartsch (2017) also note, the investigator would optimally set..."
- January 28: "Even a misspecified or dogmatic prior does not systematically bias what we learn about tau_i. This robust construction offers an opportunity to leverage tools that produce good predictions of potential outcomes even when they come with little guarantees that would otherwise ensure unbiasedness."
- February 4: "Even a misspecified or dogmatic prior does not systematically bias what we learn about tau_i. As also used in Wager et al. (2016) and Wu and Gagnon-Bartsch (2017), this robust construction offers an opportunity to leverage tools that produce good predictions of potential outcomes even when they come with little guarantees that would otherwise ensure unbiasedness."
- January 28: "When high-dimensional unit characteristics are available, machine learning offers a solution to the prediction problems implicit to unbiased estimation. Effectively, machine learning engages in automated specification searches to find a model that predicts well."
- February 4: "When high-dimensional unit characteristics are available, machine learning offers a solution to the prediction problems implicit to unbiased estimation. Effectively, machine learning engages in automated specification searches to find a model that predicts well, which Wager et al. (2016) and Wu and Gagnon-Bartsch (2017) also leverage for variance reduction in the same setting."

People revise papers all the time. People have their own writing styles. They have their own academic senses and opinions. You don't get to dictate that he should put a certain sentence in the main text or a footnote and how he should do it. This is so stupid.

3 years ago # QUOTE 6 Jab 49 No Jab !

« Previous 123…17 Next »

Recently Announced

Upcoming

Economics

Job Market

Off Topic

Job Market Wiki

Job Market Candidates

Updates

Strategically omitted references in Spiess (2017) JMP

Reply »