Non-standard errors « Economics Job Market Rumors

1 year - 234 posts - Latest - RSS
Thread: 387 Goods vs 203 No Goods

Economics Job Market Rumors » Job Market » Finance Job Rumors

Non-standard errors

Economist
476a

We were thought in UG that SEs neglect model uncertainty. We now also have a bunch of Bayesian procedures that explicitly model the model uncertainty. What is the novelty of this paper?

This is more than model uncertainty. There is latitude in deciding what efficiency or liquidity actually mean, in choosing how to measure them, in deciding the relevant sampling frequency, etc.

Ok, I can understand this. Something like uncertainty of objects that are modeled that precedes the model uncertainty?

Exactly

1 year ago # QUOTE 0 Volod 0 Vlad !
Economist
c40e

4D19 here. On further reflection, I think the main issue with the research design here is that the fincap authors’ incentives differ dramatically from those that drive actual research outcomes. Most importantly, I doubt any serious researcher is participating fincap to score a publication out of this. I am not a believer in rational agents producing public good, let alone when those agents are finance people - this applies to both participants and evaluators. There could be networking incentives, however, for which the actual research output is likely irrelevant. All these would only raise the dispersion on empirical choices and inflate evaluation scores. Overall, there is a likely huge selection bias that would only raise the non-standard errors the paper talks about. We never observe the counter factual non-participants’ outcomes who *chose* not to participate. I would be surprised if a top-5 econ journal wouldn’t desk rejects this. Finance journals might be a different story.

You missed the point of the paper completely. There is a large literature by now warning about publication bias and p-hacking. This paper shows that when authors are not incentivized to p-hack (as here is no benefit for finding significant results) there is a wide range of "correct" outcomes. At least "correct" as perceived by experienced finance researchers. This means that empirical researchers who do want to p-hack can choose any of these "correct" results, and tell the story they want to tell. Is this surprising? Not really. Is it important to document and publicize widely among the community? I think so.

I don’t think I missed the point, and I agree that the paper is talking about something important. However, to empirically show what you just claimed the paper was showing, i. e., publication incentives drivers p-hacking, and hence, empirical choices, you need to show how outcomes differe in presence versus in absence of these incentives. The paper is not showing that. A similar argument applies to how experienced researchers distinguish correct from incorrect. Your conclusion is nice, but it cannot be drawn from the paper actually does.

1 year ago # QUOTE 0 Volod 3 Vlad !
Economist
2694

4D19 here. On further reflection, I think the main issue with the research design here is that the fincap authors’ incentives differ dramatically from those that drive actual research outcomes. Most importantly, I doubt any serious researcher is participating fincap to score a publication out of this. I am not a believer in rational agents producing public good, let alone when those agents are finance people - this applies to both participants and evaluators. There could be networking incentives, however, for which the actual research output is likely irrelevant. All these would only raise the dispersion on empirical choices and inflate evaluation scores. Overall, there is a likely huge selection bias that would only raise the non-standard errors the paper talks about. We never observe the counter factual non-participants’ outcomes who *chose* not to participate. I would be surprised if a top-5 econ journal wouldn’t desk rejects this. Finance journals might be a different story.

You missed the point of the paper completely. There is a large literature by now warning about publication bias and p-hacking. This paper shows that when authors are not incentivized to p-hack (as here is no benefit for finding significant results) there is a wide range of "correct" outcomes. At least "correct" as perceived by experienced finance researchers. This means that empirical researchers who do want to p-hack can choose any of these "correct" results, and tell the story they want to tell. Is this surprising? Not really. Is it important to document and publicize widely among the community? I think so.

I don’t think I missed the point, and I agree that the paper is talking about something important. However, to empirically show what you just claimed the paper was showing, i. e., publication incentives drivers p-hacking, and hence, empirical choices, you need to show how outcomes differe in presence versus in absence of these incentives. The paper is not showing that. A similar argument applies to how experienced researchers distinguish correct from incorrect. Your conclusion is nice, but it cannot be drawn from the paper actually does.

This is a different thing. The paper is all about the space for p-hacking, not p-hacking itself. Conditioning on research teams are not doing p-hacking, their results still differ from each other, even for "high-quality" research teams after peer-review. This dispersion leaves room for p-hacking if one wants to do so. During the presentation, the authors discussed a bit what you said about what actually drives the empirical choices and they said they leave it to another paper.

From the paper:
"In sum, we believe that our study shows that NSEs add nonnegligible uncertainty to the outcome of hypotheses tests in finance. We document that NSE uncertainty is similar in magnitude as SE uncertainty. This is particularly worrisome as it creates substantial space for p-hacking (which had been scoped out here)"

1 year ago # QUOTE 3 Volod 1 Vlad !
Economist
b22b

interesting. but rhe idea is copied from sociology

1 year ago # QUOTE 2 Volod 1 Vlad !
Economist
ab3f

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

1 year ago # QUOTE 2 Volod 0 Vlad !
Economist
53eb

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

1 year ago # QUOTE 8 Volod 0 Vlad !
Economist
c40e

You keep bringing up the effect of p-hacking as if the paper has an identification of when p-hacking is present: IT DOES NOT. The paper only ASSUMES that p-hacking is present otherwise. I am not dismissing the possibility that the effect exists - I believe it probably does. Your statement reads as if results would differ even more in presence of p-hacking, and that is not conclusion, but an assumption. I am saying that the very basic econometrics says you have selection issues and you need, for example, appropriate control groups to show the selection issues are not driving your findings. There are many moving parts. If the typical finance researcher can't understand this is important for research design, then p-hacking is not really a first-order problem for this profession.

This is a different thing. The paper is all about the space for p-hacking, not p-hacking itself. Conditioning on research teams are not doing p-hacking, their results still differ from each other, even for "high-quality" research teams after peer-review. This dispersion leaves room for p-hacking if one wants to do so. During the presentation, the authors discussed a bit what you said about what actually drives the empirical choices and they said they leave it to another paper.
From the paper:
"In sum, we believe that our study shows that NSEs add nonnegligible uncertainty to the outcome of hypotheses tests in finance. We document that NSE uncertainty is similar in magnitude as SE uncertainty. This is particularly worrisome as it creates substantial space for p-hacking (which had been scoped out here)"

1 year ago # QUOTE 1 Volod 2 Vlad !
Economist
54b7

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

This. And it is not just about replicability. The paper’s results also imply that the profession should be much more open to research that revisits existing evidence and makes different design choices, regardless of the results.

1 year ago # QUOTE 2 Volod 0 Vlad !
Economist
54b7

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

This. And it is not just about replicability. The paper’s results also imply that the profession should be much more open to research that revisits existing evidence and makes different design choices, regardless of the results.

1 year ago # QUOTE 3 Volod 0 Vlad !
Economist
1f23

To me, the paper has nothing to do with the identification of p-hacking. In fact, the point of the paper is exactly the opposite, one could hide p-hacking hence not identifiable. The paper shows people can deliver different results for exactly the same hypothesis using the same data. This, to me at least, has 2 implications:
(1) One *could* P-hack if that's desired, and hide it in the range so that we never know.
(2) One could have a non-p-hacked result but someone else can conclude this result is p-hacking

You keep bringing up the effect of p-hacking as if the paper has an identification of when p-hacking is present: IT DOES NOT. The paper only ASSUMES that p-hacking is present otherwise. I am not dismissing the possibility that the effect exists - I believe it probably does. Your statement reads as if results would differ even more in presence of p-hacking, and that is not conclusion, but an assumption. I am saying that the very basic econometrics says you have selection issues and you need, for example, appropriate control groups to show the selection issues are not driving your findings. There are many moving parts. If the typical finance researcher can't understand this is important for research design, then p-hacking is not really a first-order problem for this profession.
This is a different thing. The paper is all about the space for p-hacking, not p-hacking itself. Conditioning on research teams are not doing p-hacking, their results still differ from each other, even for "high-quality" research teams after peer-review. This dispersion leaves room for p-hacking if one wants to do so. During the presentation, the authors discussed a bit what you said about what actually drives the empirical choices and they said they leave it to another paper.
From the paper:
"In sum, we believe that our study shows that NSEs add nonnegligible uncertainty to the outcome of hypotheses tests in finance. We document that NSE uncertainty is similar in magnitude as SE uncertainty. This is particularly worrisome as it creates substantial space for p-hacking (which had been scoped out here)"

1 year ago # QUOTE 4 Volod 0 Vlad !
Economist
c40e

Interesting how differently people, e.g., 122e and f2bb, interpret what the paper is about. Seems like the paper is falling into the same trap as it claims to be uncovering... I guess it is up to authors if they want to clarify this.

To me, the paper has nothing to do with the identification of p-hacking. In fact, the point of the paper is exactly the opposite, one could hide p-hacking hence not identifiable. The paper shows people can deliver different results for exactly the same hypothesis using the same data. This, to me at least, has 2 implications:
(1) One *could* P-hack if that's desired, and hide it in the range so that we never know.
(2) One could have a non-p-hacked result but someone else can conclude this result is p-hacking

You keep bringing up the effect of p-hacking as if the paper has an identification of when p-hacking is present: IT DOES NOT. The paper only ASSUMES that p-hacking is present otherwise. I am not dismissing the possibility that the effect exists - I believe it probably does. Your statement reads as if results would differ even more in presence of p-hacking, and that is not conclusion, but an assumption. I am saying that the very basic econometrics says you have selection issues and you need, for example, appropriate control groups to show the selection issues are not driving your findings. There are many moving parts. If the typical finance researcher can't understand this is important for research design, then p-hacking is not really a first-order problem for this profession.
This is a different thing. The paper is all about the space for p-hacking, not p-hacking itself. Conditioning on research teams are not doing p-hacking, their results still differ from each other, even for "high-quality" research teams after peer-review. This dispersion leaves room for p-hacking if one wants to do so. During the presentation, the authors discussed a bit what you said about what actually drives the empirical choices and they said they leave it to another paper.
From the paper:
"In sum, we believe that our study shows that NSEs add nonnegligible uncertainty to the outcome of hypotheses tests in finance. We document that NSE uncertainty is similar in magnitude as SE uncertainty. This is particularly worrisome as it creates substantial space for p-hacking (which had been scoped out here)"

1 year ago # QUOTE 2 Volod 0 Vlad !
Economist
3ebc

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

This. And it is not just about replicability. The paper’s results also imply that the profession should be much more open to research that revisits existing evidence and makes different design choices, regardless of the results.

Then, people will say this is a joke paper. Finance is not that open mind

1 year ago # QUOTE 0 Volod 0 Vlad !
Economist
92eb

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

This. And it is not just about replicability. The paper’s results also imply that the profession should be much more open to research that revisits existing evidence and makes different design choices, regardless of the results.

Then, people will say this is a joke paper. Finance is not that open mind

If the paper contributes to correcting the biases of academic Finance, that would be a huge contribution.

1 year ago # QUOTE 0 Volod 0 Vlad !
Economist
6d93

Did we really need a paper to tell us there are many “degrees of freedom” within the set of acceptable research designs and empirical measures of theoretical constructs.

Every published paper presents results from selectively chosen cells of an N-dimensional matrix of defensible research design choices.

Is this really a novel idea to fin/econ scholars? Did all fin/econ authors really think there was only one “right” way to analyze the data in a given paper, and ex post managed to convince themselves they chose that one “right” way ??

A harsh comment: but anyone who finds these results surprising is willfully ignorant of the publication production process and incentives in the field. Doesn’t mean it’s not useful to read it into the record, but not surprised in the least.

1 year ago # QUOTE 6 Volod 1 Vlad !
Economist
de93

Did we really need a paper to tell us there are many “degrees of freedom” within the set of acceptable research designs and empirical measures of theoretical constructs.
Every published paper presents results from selectively chosen cells of an N-dimensional matrix of defensible research design choices.
Is this really a novel idea to fin/econ scholars? Did all fin/econ authors really think there was only one “right” way to analyze the data in a given paper, and ex post managed to convince themselves they chose that one “right” way ??
A harsh comment: but anyone who finds these results surprising is willfully ignorant of the publication production process and incentives in the field. Doesn’t mean it’s not useful to read it into the record, but not surprised in the least.

That is pretty much the consensus in this thread. Not surprising, but interesting.

1 year ago # QUOTE 1 Volod 0 Vlad !
Economist
d702

Given the large space of p hacking, assuming researchers want to produce results that help publications and tenure, it is fair to conclude most empirical papers are useless.

Actually, this means we need to adapt scientific research process from physics and engineering, published papers and results should not be treated as fact, it should be repeatedly tested by other researchers, and having disproved results should not be shamed as p-hacking, and such validation efforts must be rewarded as well by the profession. In my opinion this is the way to go as a profession. That we have to recover the non standard errors by essentially repeat this paper on empirical research going forward.

This. And it is not just about replicability. The paper’s results also imply that the profession should be much more open to research that revisits existing evidence and makes different design choices, regardless of the results.

Then, people will say this is a joke paper. Finance is not that open mind

Then it is a j0ke field

1 year ago # QUOTE 0 Volod 0 Vlad !
Economist
2d9c

^ even not interesting

1 year ago # QUOTE 0 Volod 0 Vlad !
Economist
ea47

Did we really need a paper to tell us there are many “degrees of freedom” within the set of acceptable research designs and empirical measures of theoretical constructs.
Every published paper presents results from selectively chosen cells of an N-dimensional matrix of defensible research design choices.
Is this really a novel idea to fin/econ scholars? Did all fin/econ authors really think there was only one “right” way to analyze the data in a given paper, and ex post managed to convince themselves they chose that one “right” way ??
A harsh comment: but anyone who finds these results surprising is willfully ignorant of the publication production process and incentives in the field. Doesn’t mean it’s not useful to read it into the record, but not surprised in the least.

Yes. We needed to know that different but resonable and justifiable choices lead to dramatically different-and event contradictory-conclusions, especially given how much inertia there is in research in Finance and how reluctant the profession is to new research that questions “well established results.” We needed this debate in the profession and the paper is welcome because of that. If you want to be surprised, go to the circus.

1 year ago # QUOTE 4 Volod 2 Vlad !
Economist
6d93

"We needed to know that different but reasonable and justifiable choices lead to dramatically different-and event contradictory-conclusions"

Did you really not know this before? If not... I guess you aren't running the data on your projects. Anyone running data (should) know this.

1 year ago # QUOTE 2 Volod 1 Vlad !
Economist
b478

"We needed to know that different but reasonable and justifiable choices lead to dramatically different-and event contradictory-conclusions"
Did you really not know this before? If not... I guess you aren't running the data on your projects. Anyone running data (should) know this.

Yes. Empiricists know that in many cases you can cook up whatever results to support your story.

1 year ago # QUOTE 1 Volod 0 Vlad !

« Previous 1 234…12 Next »

Recently Announced

Upcoming

Economics

Job Market

Off Topic

Math

Job Market Wiki

Job Market Candidates

Updates

Non-standard errors

Reply »