NC p-hacked. She did not fabricate. She did what DC did but they got away with it.
We need DETAILS, CUES, WHATSOEVERYou can use some patience bruh.
NC problem runs much deeper than just herself. Glad she was able to land an industry job that pays well. Guess industry people don't really care about data fabrication.
You don't get your paper retracted for p-hacking
There is no difference between most "p-hacking" and fraud. Much of the p-hacking that goes on involves dropping inconvenient participants, DV's, or entire conditions for no reason other than obtaining a desired result. If this kind of BS is going on, then it's no different than just making up the data.
Interesting.
It does feel intuitively less wrong to drop participants without disclosure than to make up responses. Seems like a classic commission/omission asymmetry?
Maybe it’s better to characterize disclosed (but unpreregistered) dropping as p-hacking and undisclosed dropping as fraud?
Of course none of these would matter given the combination of open data/materials/preregistration (plus some way to check the authenticity of data and trace the history on Qualtrics).
When does excluding participants go from p-hacking to fraud?
If I exclude 5 "outliers" per condition, is it p-hacking or fraud?
What if I exclude 10?
If I exclude only one, but don't disclose: is it fraud?
Always report the analyses with and without the outliers, unless you pre specify exclusion criteria. Even if you specify exclusion criteria, it’s worth reporting outliers.
Interesting.
It does feel intuitively less wrong to drop participants without disclosure than to make up responses. Seems like a classic commission/omission asymmetry?
Maybe it’s better to characterize disclosed (but unpreregistered) dropping as p-hacking and undisclosed dropping as fraud?
Of course none of these would matter given the combination of open data/materials/preregistration (plus some way to check the authenticity of data and trace the history on Qualtrics).When does excluding participants go from p-hacking to fraud?
If I exclude 5 "outliers" per condition, is it p-hacking or fraud?
What if I exclude 10?
If I exclude only one, but don't disclose: is it fraud?
Always report the analyses with and without the outliers, unless you pre specify exclusion criteria. Even if you specify exclusion criteria, it’s worth reporting outliers.
Interesting.
It does feel intuitively less wrong to drop participants without disclosure than to make up responses. Seems like a classic commission/omission asymmetry?
Maybe it’s better to characterize disclosed (but unpreregistered) dropping as p-hacking and undisclosed dropping as fraud?
Of course none of these would matter given the combination of open data/materials/preregistration (plus some way to check the authenticity of data and trace the history on Qualtrics).When does excluding participants go from p-hacking to fraud?
If I exclude 5 "outliers" per condition, is it p-hacking or fraud?
What if I exclude 10?
If I exclude only one, but don't disclose: is it fraud?
But the problem is people don't. Another beautiful CB curve by the way:
https://twitter.com/cb_pcurve/status/1351140102903390212
I agree that p-hacking and fraud are very close together. But remember that p-hacking is still classified as a "QRP" rather than outright misconduct. Our current definition for misconduct (i.e., fabrication) is outdated. Journals need to push for new policies, and the recent JCR policies are a good start but not enough.
My experience is that reviewers make you drop outlier analyses. So that kind of stinks.
I’m actually less concerned about outliers, which can be easily transformed to reduce leverage, than inflated ceilings and floors with likert scales. If you have either an inflated floor or ceiling effect you are essentially poorly measuring some latent construct . WK at rice wrote a paper about something like this back in the day, but it never caught on.
Always report the analyses with and without the outliers, unless you pre specify exclusion criteria. Even if you specify exclusion criteria, it’s worth reporting outliers.Interesting.
It does feel intuitively less wrong to drop participants without disclosure than to make up responses. Seems like a classic commission/omission asymmetry?
Maybe it’s better to characterize disclosed (but unpreregistered) dropping as p-hacking and undisclosed dropping as fraud?
Of course none of these would matter given the combination of open data/materials/preregistration (plus some way to check the authenticity of data and trace the history on Qualtrics).When does excluding participants go from p-hacking to fraud?
If I exclude 5 "outliers" per condition, is it p-hacking or fraud?
What if I exclude 10?
If I exclude only one, but don't disclose: is it fraud?But the problem is people don't. Another beautiful CB curve by the way:
https://twitter.com/cb_pcurve/status/1351140102903390212
I agree that p-hacking and fraud are very close together. But remember that p-hacking is still classified as a "QRP" rather than outright misconduct. Our current definition for misconduct (i.e., fabrication) is outdated. Journals need to push for new policies, and the recent JCR policies are a good start but not enough.
excluding participants without disclosing it or doing so equally across conditions is fraud.
p-hacking happens when you first try excluding people who have taken the survey before, then try excluding the ones who failed an attention test, etc. If you run enough analyses based on justifiable exclusion criteria, then you increase your chances of getting a desired result. Dropping even one participant who did X in one condition without dropping a participant who also did X in the other is fraud.
p-hacking increases your chances of finding an effect but it has limitations.
When does excluding participants go from p-hacking to fraud?
If I exclude 5 "outliers" per condition, is it p-hacking or fraud?
What if I exclude 10?
If I exclude only one, but don't disclose: is it fraud?
You can drop 90% of your sample if you want to as long as you disclose it. That way reviewers can and probably will question you about it, so you'll want to be able to justify it and ideally have pre-registered it to address their concerns. So it's not about how many observations you drop. It's about doing so equally across conditions and being transparent about it so that it's not fraud. Pre-registering helps address whether any given result is p-hacked (justified only after having seen if a certain action helps) or not. P-hacking is motivated reasoning, not fraud, and it's not limited to behavioral research. Deciding and justifying what should go into a regression or a model after you see how it affects your results is also p-hacking.
In the long term, how do you get rid of the incentive to cheat, that should be the question. Here are some ideas:
1. Reduce the pressure to publish numbers, but focus on quality and content of papers. The numbers pressure is there in the job market and at tenure, because PEOPLE COUNT! If you do not count but read papers, and refuse to see a candidate with 2As as better than a candidate with none, then you will always have these problems. In the hindsight, these papers were impossible to believe.
2. Welcome and expect some null results -- bring it up as an issue when the results are all magically working well
I am all for the long-term plans, but these sort of things have been discussed time and time again in the past 10 years with not much outcome. I think vague long-term thinking derail what we should be doing at the moment - short-term, concrete plans to discourage p-hacking.
This is why I applaud the Twitter CB Curve account. I am sure it is getting on the p-hackers' nerves, because the p-curve results gets embedded onto the Altmetric on the respective article's website. Hahaha
nonsense. null effects is no research just as fraud is no research just as no research s no research. reward only good research and plenty of it. if you want to publish nulls, don't d-hack i.e., be upfront and don't hide all the dum@$$ things you did that got you the null.
2. Welcome and expect some null results -- bring it up as an issue when the results are all magically working wellI am all for the long-term plans, but these sort of things have been discussed time and time again in the past 10 years with not much outcome. I think vague long-term thinking derail what we should be doing at the moment - short-term, concrete plans to discourage p-hacking.
This is why I applaud the Twitter CB Curve account. I am sure it is getting on the p-hackers' nerves, because the p-curve results gets embedded onto the Altmetric on the respective article's website. Hahaha
I just love this guy. He is unafraid to make enemies with very senior people in psychology.
https://replicationindex.com/2021/01/19/personalized-p-values/
Lots of marketing professors on the bottom of this list.Look at Kahneman.
P-hack your way to nobel prize I guess