Saturday, 5 August 2017

Why I hate the ‘tone debate’ in psychology and you should too

There is an extremely boring, seemingly indestructible, debate going on in psychology about the ‘tone’ of scientific debate in our field. The root of this peculiar argument is that the nature of some discussion surrounding open science and reform is apparently too negative, too critical, and too personal, and that this pushes people away from doing better science.

The entire debate is so nauseatingly self-absorbed. It really pains me to see so many smart colleagues who I respect and admire devoting so much of their time to being concerned about this issue. Hey, it’s their time, who am I to judge. But what a waste it is of our collective energy, and I wish people could see why.

A few unfiltered thoughts, and then I will never speak of this again.

1. There’s this idea that open science will attract more 'disciples' if it comes across as having a more positive, inclusive tone. Goodness me, what a load of honking bullshit this is. Open science will attract individual adopters for three reasons: (1) when scientists grow a conscience and appreciate that their public mission demands transparency and reproducibility; (2) when scientists decide to take advantage of individual incentives (career and social) for being open (e.g. Registered Reports, joining SIPS etc.); (3) when funders, journals and institutions make it a requirement. All of these are in progress. The cold embrace of open science by gatekeepers and regulators is in the post – it is only a matter of time before transparent, reproducible practices will be required if you want to spend public money. That’s why I tell early career researchers to get ahead now because the ground is shifting under your feet, and I am one of the people shifting it. What ‘tone’ we set in public discourse about science (and open science) is thoroughly irrelevant to all of this; indeed, thinking that this changes anything is nothing more than hubristic navel gazing.

2. Stop seeing open science as some kind of evangelist movement that needs to save souls from damnation. Yes it is a political movement. Yes, the more people adopt it the better. But you won’t get them to do it through the tone of critical discussion or by preaching from the pulpits. We aren’t desperate god botherers in suits waving leaflets at people on the street, and we aren’t leading the congregation in prayer. We get scientists to adopt open practices by changing the system to make openness attractive and essential.  

3. Leave Satire The Fuck Alone. If we are so thin skinned that we can’t laugh at ourselves from time to time (and, jesus, even learn something from it), we are already lost. 

4. Don’t fall into the sanctimonious trap of being concerned about tone on behalf of others. No, you are not the voice of early career researchers or the vulnerable or Nixon's silent majority. All of these communities can speak for themselves, and they do, especially in the flat landscape of social media. Yes, your crappy anecdotes about some researcher who really wanted to be open but was put off by a sharp twitter discussion count for precisely dick.

5. Psychology is a discipline that has no problem praising the Living Fuck out of individuals when they publish some stunning result or get a big grant. Hey guess what: critique follows the same rules. You live by the sword, you die by it.

6. Ignoring the ‘tone debate’ isn’t the same as turning a blind eye to abuse. There is painfully little evidence that any of the much touted tone infractions constitute abuse, or that abuse in the open science community is any greater than the baseline in academia. And guess what, it isn’t the job of the open science community to solve all problems in academia.

7. Follow Wheaton’s Law if you prefer but don’t let anything stand in the way of delivering the criticism that is required in service of your mission. Sometimes this will require setting a tone that risks causing offence. That's ok, so long as it is in service of the mission. I say this repeatedly, but remember who you serve. It isn’t yourself and it isn’t your colleagues. It isn’t early career researchers or vulnerable scientists, or scientists without a lot of resources. Science serves the public who pay for science. Ask yourself how our snivelling, embarrassing debates about ‘tone’ appear to the people who fund the work we do. Ask yourself how pathetic this all looks and why other sciences aren’t doing it. Then snap out of it.

8. Finally, and this is the kicker: Discussions about ‘tone’ were originally orchestrated (rather shrewdly, it must be said) by opponents of reform as a way of diverting attention from systemic changes to science that (a) threaten the organic structures that underpin their own power and (b) would expose their own sloppy research practices. We are in an intensely political period and whenever you argue about tone you are playing your opponents’ game by your opponents’ rules. While we tear ourselves sideways worrying about such trival nonsense, they are smirking from their thrones.

This is the last thing I will ever say about the ‘tone’ debate. We need to change the way psychological science works, not the way some people talk about psychological science.

Thursday, 27 July 2017

Open-ended, Open Science

In this special guest post, Rob McIntosh, associate editor at Cortex and long-time member of the Registered Reports editorial team, foreshadows a new article type that will celebrate scientific exploration in its native form.

Exploratory Reports will launch next month and, now, we need your input to get it right.

Chris has kindly allowed me to crash his blog, to publicise and to gather ideas and opinions for a new article type at Cortex. The working name is Exploratory Reports. As far back as 2014, in his witterings and twitterings, Chris trailered a plan for Cortex to develop a format for open-ended science, a kind of louche, relaxed half-cousin to the buttoned-up and locked-down Registered Reports. Easier tweeted than done. We are now preparing to launch this brave new format, but even as we do so, we are still wrestling with some basic questions. Does it have a worthwhile role to play in the publishing landscape? Can it make a meaningful contribution to openness in science? What should its boundaries and criteria be? And is there a better name than Exploratory Reports?

Visitors to this blog will have a more-than-nodding familiarity with misaligned incentives in science, with the ‘deadly sin’ of hidden flexibility, and with the damage done to reliability when research conducted in an open-ended, see-what-we-can-find way, is written into the record as a pre-planned test of specific hypotheses. No one doubts that exploratory research has a vital role to play in empirical discovery and hypothesis generation, nor that it can be rigorous and powerful (see recent blog discussions here and here). But severe problems can arise from a failure to distinguish between exploratory and confirmatory modes of enquiry, and most perniciously from the misrepresentation of exploratory research as confirmatory.

A major driver of this misrepresentation is the pervasive idealisation of hypothesis-testing, throughout our scientific training, funding agencies, and journals. Statistical confirmation (or disconfirmation) of prior predictions is inferentially stronger than the ‘mere’ delineation of interesting patterns, and top journals prefer neat packages of strong evidence with firm impactful conclusions, even if our actual science is often more messy and… exploratory. Given a more-or-less-explicit pressure to publish in a confirmatory mode, it is unsurprising that individual scientists more-or-less-wittingly resort to p-hacking, HARKing, and other ‘questionable research practices’.

Regulars of this blog will need no further education on such QRPs, or on the mighty and multi-pronged Open Science movement to reform them. Still less will you need reminding of the key role that study pre-registration can play by keeping researchers honest about what was planned in advance. Pre-registration does not preclude further exploration of the data, but it keeps this clearly distinct from the pre-planned aspects, eliminating p-hacking, HARKing, and several other gremlins, at a stroke. The promise of enhanced truth value earns pre-registered studies an Open Practices badge at a growing number of journals, and it has even been suggested that there should be an automatic bonus star in the UK Government’s Research Excellence Framework (where stars mean money).

This is fine progress, but it does little to combat the perceived pre-eminence of confirmatory research, one of the most distorting forces in our science. Indeed, a privileged status for pre-registered studies could potentially intensify the idealisation of the confirmatory mode, given that pre-registration is practically synonymous with a priori hypothesis testing. A complementary strategy would therefore be for journals to better value and serve more open-ended research, in which data exploration and hypothesis generation can take precedence over hypothesis-testing. A paper that is openly exploratory, which shows its working and shares its data, is arguably as transparent in its own way as a pre-registered confirmatory study. One could even envisage an Open Practices badge for explicitly exploratory studies. 

Some journal editors may believe that it is typically inappropriate to publish exploratory work. But this is not the case at Cortex, where the field of study (brain-and-behaviour) is relatively uncharted, where many research questions are open-ended (e.g. What are the fMRI or EEG correlates of task X? What characterises patient group Y across test battery Z?), and where data collection is often costly because expensive technologies are involved or a rare or fleeting neuropsychological condition is studied. It is hard to estimate how much of the journal’s output is really exploratory because, whilst some authors have the confidence to make exploratory work explicit, others may still dress it in confirmatory clothing. If publication is their aim, then they are wise to do so, because the Action Editor or reviewers could be unsympathetic to an exploratory approach.

Hence, a new article type for exploratory science, where pattern-finding and hypothesis generation are paramount, and where the generative value of a paper can even outweigh its necessary truth value. A dedicated format is a commitment to the centrality of exploratory research in discovery. It also promotes transparency, because the incentives to misrepresentation are reduced, and the claims and conclusions can be appropriate to the methods. Some exploratory work might provide strong enough evidence to boldly assert a new discovery, but most will make provisional cases, seeding testable hypotheses and predictions for further (confirmatory) studies. The main requirements are that the work should be rigorous, novel, and generative.

Or that is the general idea. The devil, as ever, is in the detail. Will scientists – as authors, reviewers and readers - engage with the format? What should exploratory articles look like, and can we define clear guidelines for such an open-ended and potentially diverse format? How do we exercise the quality control to make this a high-status format of value to the field, not a salvage yard for failed experiments, or a soapbox for unfettered speculation? Below, a few of the questions keeping us awake at night are unpacked a little further. Your opinions and suggestions on these questions, and any aspect of this venture, would be most welcome. 

1. Scope of the format. At the most restrictive end, the format would be specific for studies that take an exploratory approach to open-ended questions. Less restrictive definitions might allow for experimental work with no strong a priori predictions, or even for experiments that had prior predictions but in which the most interesting outcomes were unanticipated. At the most inclusive end, any research might be eligible that was willing to waive all claims dependent upon pre-planning. Are there clear boundaries that can be drawn? 

2. Exploration and review. A requirement for submission to this format will be that the full data are uploaded at the point of submission, sufficient to reproduce the analyses reported. To what extent should reviewers, with access to the data, be allowed to recommend/insist that further analyses, of their own suggestion, should be included in the final paper? 

3. Statistical standards. Conventional significance testing is arguably meaningless in the exploratory mode, and it has even been suggested that this format should have no p-values at all. There will be a strong emphasis on clear data visualisation, showing (where feasible) complete observations. But some means of quantifying the strength of apparent patterns will still be required, and it may be just too radical to exclude p values altogether. When using conventional significance testing, should more stringent criteria for suggestive and significant evidence be used? More generally, what statistical recommendations would you make for this format, and what reporting standards should be required (e.g. confidence intervals, effect sizes, adjusted and non-adjusted coefficients etc.)? 

4. Evidence vs. theory. Ideally, a good submission presents a solid statistical case from a large dataset, generating novel hypotheses, making testable predictions. The reality is often liable to be more fragmentary (e.g. data from rare neuropsychological patients may be limited, and not easily increased). Can weaker evidence be acceptable in the context of a novel generative theoretical proposal, provided that the claims do not exceed the data? 

5. The name game. The working title for this format has been Exploratory Reports. The ambition is to ‘reclaim’ the term ‘exploratory’ from a slightly pejorative sense it has acquired in some circles. Let’s make exploration great again! But does this set up too much of an uphill struggle to make this a high-status format; and is there anyway a better, fresher term (Discovery Reports; Open Research)?

Rob will oversee this new article format when it launches next month. Please share your views in the comments below or you can feed back directly via email to Rob or Chris, or on twitter.

Thursday, 1 June 2017

The Saboteur

“You’re our insurance policy”, DC had said. “Our last chance. If the worst happens, you are going to drive a bomb into Brexit”.

“If Remain wins, you win – I’ll quit before the next GE and you’ll stand as PM. And if Remain loses, you will still win, for a time, and you'll be an even bigger hero in the long run.” He explains the plan. You accept.

You lay low during the referendum campaign. A few low-key speeches for Remain. Nothing grandiose. Nothing to alienate the Brexiteers.

Then it happens. Leave somehow wins. Boris and Michael, ashen-faced, accept victory as though they’ve been given a week to live.

The Greek tragedy unfolds on cue. Boris and Michael murder each other as they were always destined to. Andrea insults you for being childless. Amateur hour.

You step carefully over their quivering corpses to ascend the throne.

“You’ll keep it simple”, DC explained. “Just slogans and gumpf. Nothing for months. Put the Brexiteers in charge of the ship. They'll flounder all by themselves because the task is impossible”.

You position Boris, Liam and David at the helm. Even competent ministers would struggle, but they are so hopeless you can’t possibly fail.

"Brexit means Brexit". Say it repeatedly. Perfectly vacuous and wastes time while sounding decisive.

A court challenge is issued. Good. You fight it which plays well with the hate rags. At best the courts will weaken Brexit and it chews up even more time.

The courts fail. Annoying. You proceed to the next phase.

“You'll have to push the hardest possible Brexit.”, DC had explained. “You have to neuter UKIP while giving the opposition every chance to weaken it in the Commons and Lords.” 

Corbyn goes along with you. Didn't expect that at all. You handed him soft Brexit on a platter and he batted it away. Always was a Leaver, the silly old fool.

You realise you have the unpleasant task now of issuing Article 50 but it doesn’t matter. Plays well with the papers and the lawyers say it’s revocable anyway. Pure stage show.

“If you get to this point”, said DC, “you might be pushed to more extreme measures. The party will take a hit but we’ll recover - we're the most successful party in the world, afterall.”

You call an election, going back on a promise but this plays well into the narrative you need to create. Brexit on pause at least. Your goal is the darkest and most hidden of secrets: to throw the vote but only just. Political microsurgery.

Nobody knows but you and the tightest of inner circles. It’s an election that can't be lost but you need to win by a small enough margin to derail the train.

“Strong and stable”. Perfect. Easy pickings for opponents who will flip it to "weak and wobbly". Yes. You can play this part.

“If you have to force an election, you’ll need to play the coward and liar to a tee”, DC had warned. “It will hurt, but remember, at the end of all this you’ll go down in history”.

You line up the dominoes. A weak manifesto. A dementia tax to alienate your core older vote. Devoid of costings while attacking opponents for their costings. Double standards galore, and ample rope for the left to hang you with.

The vote is still too high. Reverse a manifesto promise during the campaign. It's never been tried but you need this to be close. Issue angry denials. Refuse to answer direct questions. Refuse to attend debates. 

Refuse to engage at all with voters. Appear bad tempered, limp and incompetent in media interviews. Some will say it looks like you don't want to win but they will never realise just how true that is.

You hate the election campaign but the narrative is working. Corbyn has stepped up at last. The polls are converging. You’ll still win but with luck, you’ll have weakened the Brexiteer’s hands just enough.

Election day comes. Corbyn does well. He plays best when backed into a corner. You still win the most seats but SNP ensures that it’s a hung parliament.

Finally. You can breathe.

Brexit on hold, maybe forever. Soft Brexit at most. The Scots will now vote to remain in the UK. And by now public opinion favours remaining in the EU anyway. You resign. Bomb detonated, agreement with DC met.

Corbyn forms a struggling minority government but he won't last long. He has too many enemies within and your party will be back in charge by 2022.

You leave politics, offering up health reasons.

Now the real work begins: the diaries. You recorded everything. Everything. Every stupid meeting with Boris. Every ridiculous phone call with The Donald. It will be the most explosive political manifesto ever written - how you destroyed the far right from within your own party. They will celebrate you as the hero who saved Britain from Brexit and the UK from Balkanization.

Disclaimer: no I haven't donned a tin foil hat - this is indeed a work of pure fiction. Wouldn't it be entertaining though!

Thursday, 13 April 2017

Seven questions about my new book: the Seven Deadly Sins of Psychology

So I wrote a short book about psychology and the open science movement (HOLY CRAP IS THIS REALLY HAPPENING)


Allow me to compose myself. Yes, yes it is.

The book is called The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice. You can find it at Amazon or Princeton University Press.

The illustration at the top of this post is a mosaic of the extraordinary artwork in the book, created by the richly talented Anastasiya Tarasenko (you can see more of her work here, and she is on twitter too). Each woodcut depicts one of the deadly sins, and there is one virtue as well, making up the eight chapters. I was inspired to pursue this general concept by the imagery in a film called the Ninth Gate in which bookdealer Dean Corso goes searching for a book written by the Devil. The final illustrations here are a marriage between that general concept and Anastasiya's creative genius. 

My friend Pete Etchells said I was rubbish at promoting my book, so I've decided to post a 7-part Q&A about it. Um yeah, I realise this is just me asking myself questions, but I'm a bit of a bastard, even to myself.


1. Why did you decide to write this book?
Over the last fifteen years I‘ve become increasingly fed up with the “academic game” in psychology, and I strongly believe we need to raise standards to make our research more transparent and reliable. As a psychologist myself, one of the key lessons I’ve learned is that there is a huge difference between how the public thinks science works and how it actually works. The public have this impression of scientists as objective truth seekers on a selfless mission to understand nature. That’s a noble picture but bears little resemblance to reality. Over time, the mission of psychological science has eroded from something that originally was probably quite close to that vision but has now become a contest for short-term prestige and career status, corrupted by biased research practices, bad incentives and occasionally even fraud.

Many psychologists struggle valiantly against the current system but they are swimming against a tide. I trained within that system. I understand how it works, how to use it, and how it can distort your thinking. After 10 years of “playing the game” I realised I didn’t like the kind of scientist I was turning into, so I decided to try and change the system and my own practices – not only to improve science but to help younger scientists avoid my predicament. At its heart this book lays out my view of how we can reinvigorate psychology by adopting an emerging philosophy called “open science”. Some people will agree with this solution. Many will not. But, above all, the debate is important to have.

2. It sounds like you’re quite skeptical about science generally
Even though I’m quite critical about psychology, the book shouldn’t be seen as anti-science – far from it. Science is without doubt the best way to discover the truth about the world and make rational decisions. But that doesn’t mean it can’t or shouldn’t be improved. We need to face the problems in psychology head-on and develop practical solutions. The stakes are high. If we succeed then psychology can lead the way in helping other sciences solve similar problems. If we fail then I believe psychology will fade into obscurity and become obsolete.

3. Would it matter if psychology disappeared? Is it really that important?
Psychology is a huge part of our lives. We need it in every domain where it is important to understand human thought or behaviour, from treating mental illness, to designing traffic signs, to addressing global problems like climate change, to understanding basic (but extraordinarily complex) mental functions such as how we see or hear. Understanding how our minds work is the ultimate journey of self-discovery and one of the fundamental sciences. And it’s precisely because the world needs robust psychological science that researchers have an ethical obligation to meet the high standards expected of us by the public. If, to some of my colleagues, that sounds rather high-handed and moralistic, well, it is. Suck it up guys.

4. Who do you think will find your book most useful?
I have tried to tailor the content for a variety of different audiences, including anyone who is interested in psychology or how science works. Among non-scientists, I think the book may be especially valuable for journalists who report on psychological research, helping them overcome common pitfalls and identify the signs of bad or weak studies. At another level, I’ve written this as a call-to-arms for my fellow psychologists and scientists in closely aligned disciplines, because we need to act collectively in order to fix these problems. And the most important readers of all are the younger researchers and students who are coming up in the current academic system and will one day inherit psychological science. We need to get our house in order to prepare this generation for what lies ahead and help solve the difficulties we inherited.

5. So what exactly are the problems facing psychology research?
I’ve identified seven major ills, which (a little flippantly, I admit) can be cast as seven deadly sins. In order they are Bias, Hidden Flexibility, Unreliability, Data Hoarding, Corruptibility, Internment, and Bean Counting. I won’t ruin the suspense by describing them in detail, but they all stem from the same root cause: we have allowed the incentives that drive individual scientists to fall out of step with what’s best for scientific advancement. When you combine this with the intense competition of academia, it creates a research culture that is biased, closed, fearful and poorly accountable – and just as a damp bathroom encourages mould, a closed research culture becomes the perfect environment for cultivating malpractice and fraud.

6. It all sounds pretty bad. Is psychology doomed?
No. And I say this emphatically: there is still time to turn this around. Beneath all of these problems, psychology has a strong foundation; we’ve just forgotten about it in the rat race of modern academia. There is a growing movement to reform research practices in psychology, particularly among the younger generation. We can solve many problems by adopting open scientific practices – practices such as pre-registering study designs to reduce bias, making data and study materials as publicly available as possible, and changing the way we assess scientists for career advancement. Many of these problems are common to other fields in the life sciences and social sciences, which means that if we solve them in psychology we can solve them in those areas too. In short, it is time for psychology to grow up, step up, and take the lead.

We'll know we've succeeded in this mission when our published results become reliable and repeatable. As things currently stand, there is a high chance that any new result published in a psychology journal is a false discovery. So we’ll know we’ve cracked these problems when we can start to believe the published literature and truly rely on it. When this happens, and open practices become the norm, the closed practices and weak science that define our current culture will seem as primitive as alchemy.

7. What's wrong with your book?
Probably a lot, and that is for the community to judge. As a matter of necessity, any digestible perspective on this issue is going to contain a lot of the author's personal views. Much of what I've written stems from my own experience training up and working in science, and reasonable people can disagree about the nature of the problems and the best solutions. The field is also moving very quickly, which makes writing a book particularly challenging. On a more specific note, I have a public errata listing misprints that weren't quite caught in time before the first batch of publishing. If you bought a first edition you may come across some or all of these (a rather optimistic economist friend of mine advises you to hang on to those first edition copies, because you never know what they might be worth some day...yes...well...). And of course, if you find an additional one, let me know and it shall be amended at the next available opportunity.

I hope you enjoy it.

Thursday, 16 March 2017

Should all registered clinical trials be published as Registered Reports?

Last night I asked you brilliant folks to give me your strongest counterarguments to the following proposition: that all registered clinical trials should be published in journals only if submitted as Registered Reports (RRs).

It was a really interesting discussion, and thanks to everyone who engaged. Here are the reasons that were put forward. 

My tl;dr verdict: I’m still waiting for a good reason!

1. RRs require presenting more methodological detail than standard clinical trial registration and so expose authors to a potential competitive disadvantage.
This isn’t really a scientific objection (or least, it’s a very weak scientific objection) but I understand the strategic argument. My response is that if all registered trials have to published as RRs then everyone faces the same disadvantage, so there is no relative disadvantage.

2. Clinical trial registration is sufficient for controlling bias.
It’s not. Around 50% of clinical trials never report results, ~14% are registered after data collection is complete, and somewhere between 30-85% engage in hidden outcome switching. With depressing statistics like this, how can standard trial registration be seen as anything close to sufficient?

3. OK, clinical trial registration used to be insufficient but it’s sufficient now because requires authors to specify a primary outcome measure.
Still nope. The COMPARE project finds that authors routinely engage in hidden outcome switching even when primary outcome measures are specified. There is no logical reason why requiring something that already fails to prevent hidden outcome switching should prevent hidden outcome switching.

4. Ok fine, but a signed declaration at submission that outcomes haven’t been switched would solve hidden outcome switching.
It would probably have some effect, but then it remains easy to specify an outcome sufficiently vaguely to enable one of several variables to be cherry picked as the primary outcome measure, and so allow researchers to tick this box even when they switched. And even if this measure did reduce outcome switching, it would not reduce publication bias. RRs reduce both hidden outcome switching and publication bias. So why should any kind of declaration be preferable to all registered clinical trials being published as RRs?

5. Small companies often live or die by the results of trials. The RR model presents a risk to their livelihoods if they have to publicly admit that an intervention failed to work.
The model suggested here applies only to registered trials. If companies want to do their own internal unregistered trials and choose what to publish (where they can) based on the results, that’s up to them. The argument here is that the price of attaining credibility within the pages of a reputable peer-reviewed journal should be to register the trial as a RR.

6. RRs involve one paper arising per protocol. But a single protocol may need to produce multiple papers addressing different questions. This is also important to support the careers of early career researchers.
This sounds to me like an argument for salami slicing in the interests of careerism. But I accept that in the reality of academia, careers matter. My initial reaction is that if the research question and method are complementary enough to go in the same protocol, why aren’t the results complementary enough to go in the same paper? The easy solution to this is to separate protocols that address different questions into different RRs. That way there are as many papers to publish as there are separate research questions.

7. The RR model doesn’t force authors to publish their results. Therefore there is no guarantee that it will prevent publication bias.
This is the strongest objection so far, but even so it is virtually guaranteed to be less of a problem for RRs than under the status quo. Authors of RRs are indeed free to withdraw their papers after the protocol is provisionally accepted, but doing so triggers the publication of at least part of the registered protocol plus a reason for the withdrawal. So what is an author going to say, that they withdrew their peer-reviewed RR because the trial outcomes were negative? I suspect the research community (including the reviewers who invested time in assessing the protocol) would take a dim view on such a strategy, and it is probably for this reason that there has yet to be a single case of a withdrawn RR at any journal. In any case, if this were considered to be a serious risk, it would be straightforward to strengthen the RR model for clinical trials so that it requires authors to publish the results. If every journal published registered clinical trials only as RRs, and all RRs bore this mandate, virtually all registered trials would be published.

So that’s seven reasons, none of which I think are particularly strong.

Got anything stronger?

Sunday, 13 November 2016

Why science – and open science – matters now more than ever

The internet is abound with thinkpieces about the election of Donald Trump, and I won’t bore you with my own views, aside from one: I hear a lot of scientists saying that their work is the least important thing now; that science has slipped down their personal agenda; that there are bigger fish to fry. 

I can understand why many of us feel this way, particularly in the immediate aftermath of such a calamitous election, but it’s mistaken.

Science matters now more than ever. Despair is an indulgence we cannot afford.

There are two existential threats posed by Trump – one that I think is mostly hyperbole, and another that is deadly and very real. The first is nuclear holocaust; the second is climate change. The threat of nuclear war is a Hollywood fantasy that will almost certainly never happen because all sides know that it guarantees mutual, permanent annihilation. The second is globally disastrous and nearly certain if the world doesn’t commit to mass action. If Trump withdraws the US from the Paris agreement, and global temperatures continue to rise, then the consequences for life on Earth are appalling. If temperatures rise by just 6 degrees then most life on Earth will become extinct, including us. None of the shit we think is important now nothing will matter a damn.

This really brings home for me why open science and evidence-based policy are so vital. Science must be open to be reproducible and, in turn, to have any chance of being valued in politics. And policy based on evidence is our ticket – our one ticket – to survival. I know now that attempting to make these normative will be my life's work in academia. Every day I will focus my efforts even more on our ongoing projects to promote open science and the use of evidence in policy-making.

To all scientists: now is the time to go all in and make your work as open as possible – if not for yourself then for your children and for their children. The more scientists who adopt open practices, the more normative they will become and the better chance humans have to still be here in a century’s time.

Before Brexit and Trump we could perhaps think that the wisdom of crowds would prevail and civilisation would proceed inexorably toward a Star Trek future. We now know that this is bullshit. The wisdom of crowds is bullshit. Civilisation is fragile, a thin veneer hard fought for and easily shattered.

It doesn’t matter if your work is in a field that you consider relevant to global problems like climate change – we can never predict which areas of science might one day form the basis of a solution to a problem. Adopting open practices across as many fields as possible sends the message that science needs to be transparent, reproducible and usable by anyone.

So, please, if you are not already doing so, publicly archive your data, materials and code, pre-register your hypotheses and analysis plans, and publish your work through open access routes. Focus on getting your science right rather than hacking your way to glam papers, climbing the greasy pole, or building your empire. Recognise your own bias: don't confuse the window to the natural world with a mirror. Be vocal in communicating your work in schools and to the public, particularly outside your echo chambers, to the people in your community who are the most skeptical or suspicious about science. Be politically active in highlighting the importance of evidence-based policy. Science funding is likely to become increasingly limited so make your science count. As scientists we are indoctrinated with the notion that competition matters but it doesn't. Cooperation is far more important, and we are all on the same team now. Team survival.

Whatever field you work in, your work counts and so does your voice. The world needs to survive at least four years of Donald Trump, and hell knows whatever worse may follow him.

I look at my daughter, who is ten months old, and I realise that she and her children will be the ones who will live with the choices we make today. She is why open science matters.

Wednesday, 9 November 2016

An "Accountable Replication Policy" at Royal Society Open Science

tl;dr: we've drafted a new journal policy for increasing the publication of replication studies and we need your help to get it right.

Many readers will be aware of Sanjay Srivastava's 2012 proposal for a Pottery Barn rule to support replications at scientific journals. The basic idea is that journals should commit to publishing replication attempts of previous studies published within the journal (i.e. "you publish it, you buy it"). As Sanjay wrote at the time: "Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome."

I have always felt that this was a brilliant idea - truly one of the most innovative and concrete proposals for not only increasing the volume of replication studies but also incentivizing journals to publish original work that is more replicable in the first place. It's a little disappointing that no journal has yet taken up the policy.

That is about to change. Essi Viding and I have written a fully specified Accountable Replications Policy, to be implemented at the journal Royal Society Open Science where we both serve as editors. 

We plan to capture the core features of Sanjay's proposal while also extending it in two ways.

First, the commitment to publish replications of previous work will include not only articles published within Royal Society Open Science but in more than 20 other major psychology and neuroscience journals. 

Second, the Accountable Replications mechanism will be open to both completed replication studies (unlocking the file drawer of hitherto unpublished replications) AND to proposals for yet-to-be completed replications. In the former case, the manuscripts will be initially stripped of results and reviewed "results-blind". In the latter case, submissions will be assessed via the journal's existing Registered Report route. In both cases, the decision to publish will be based on peer review before results are known, immunising against publication bias. 

Our draft policy has been approved by the journal editorial board and Royal Society. Drafting it was a tricky process and we are now seeking to further improve it based on your views. Is the policy fit for purpose? Is it clear? Have we missed any risks or opportunities? Would you use it? Please let us know what you think in the comments below, on social media, or by email


Policy Overview   

We believe that when a journal publishes an empirical study it assumes a degree of accountability for the replicability of that study. This responsibility is reflected by the extent to which the journal commits to publishing attempted replications of previous studies that appear within its pages.

The Accountable Replication Policy commits the Psychology and Cognitive Neuroscience section of Royal Society Open Science to publishing replications of studies previously published within the journal. Authors can either submit a replication study that is already completed or a proposal to replicate a previous study. To ensure that the review process is unbiased by the results, submissions will be reviewed with existing results initially redacted (where applicable), or in the case of study proposals, before the results exist. Submissions that report close, clear and valid replications of the original methodology will be offered in principle acceptance, which virtually guarantees publication of the replication regardless of the study outcome.

The diagram below illustrates the editorial process and manuscript handling pipeline (and see here for a higher resolution image).

Detailed Policy

1.   Royal Society Open Science commits to publishing any methodologically sound attempt to replicate any study that has been published within the Psychology and Cognitive Neuroscience section of the journal, as well as any study within the remit of the Psychology and Cognitive Neuroscience section that has been published in any one of the following journals: Biological Psychiatry, Brain, Cerebral Cortex, Cortex, Cognitive Psychology, Cognition, Current Biology, European Journal of Neuroscience, Journal of Cognitive Neuroscience, any journal within the Journal of Experimental Psychology group, Journal of Experimental Social Psychology, Journal of Neuroscience, Journal of Personality and Social Psychology, Nature, Nature Neuroscience, NeuroImage, Neuron, PLOS Biology, Proceedings of the National Academy of Sciences of the USA, Psychological Science, Science, Social Psychological and Personality Science, and any journal published by the Royal Society. We will also consider, on a case-by-case basis, pre-submission enquiries for replications of studies published in journals additional to those listed here.

2.   Under this policy, authors have the choice between two routes for consideration of submissions:  a “results-blind” track in which the introduction and method of a completed study are peer reviewed while the results and discussion are temporarily withheld, or the Registered Reports track where the study protocol is reviewed and pre-accepted before the commencement of data collection.

3.   Results-blind track

3.1.  This track is intended for authors who have already completed a direct (close) replication attempt of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or in any other journal approved on a case-by-case basis by the editors. Authors should initially submit a partial manuscript containing the abstract, introduction and method sections, with the results and discussion redacted together with any other information (e.g. in the abstract) that states or implies the research outcomes. The method section must include a comprehensive and detailed account of all experimental procedures (including data inclusion and exclusion criteria) and analyses undertaken. The later Stage 2 submission cannot include the outcomes of any analyses that are not stated in the Stage 1 submission at the point of in principle acceptance. The Stage 1 manuscript should be submitted under the standard research article category.

3.2.  At Stage 1 the submission is considered by reviewers who assess the content according to the following Primary and Secondary criteria:

Stage 1 Primary Criterion #1: Whether the authors provide a sufficiently clear and detailed description of the methods for another researcher to replicate exactly the proposed experimental procedures and analysis pipeline, and to prevent undisclosed flexibility in the experimental procedures or analysis pipeline. 

Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability. 

Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses. 

Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline. 

Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.

Of the above Stage 1 criteria, editorial decisions will be based solely on the Primary Criteria. In some cases, the published methods of the original study may not provide enough detail to permit a close replication (Primary Criterion #1), and the original authors of the study may also be unwilling or unable to provide the necessary missing information. In such cases, appropriate estimates of the original procedures will be accepted provided the authors can establish that the chosen method is a reasonable estimation of the original approach, and provided they can supply documentary evidence (such as emails) proving any attempts to obtain such information from the original authors.

With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.

For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).

For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).

For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.

These contingencies are summarised in the table below (and see also the workflow diagam above).

Threat level of methodological concerns under Stage 1 Secondary Criteria #1, #2 or #3

Original study published in Royal Society Open Science
Mandatory discussion of the concern in final paper by authors
Publication of cautionary note by editors
Authors encouraged (but not required) to pre-register an additional corrective replication attempt
Authors required to conduct a corrective replication attempt to achieve Stage 1 acceptance
Original study not published in Royal Society Open Science
Mandatory discussion of the concern in final paper by authors
Publication of cautionary note by editors
Authors encouraged but not required to pre-register an additional corrective replication attempt
Authors required to conduct a corrective replication attempt to achieve Stage 1 acceptance

Other than the scenario described above for severe concerns, Stage 1 manuscripts will be rejected outright only if reviews indicate that the replication attempt failed to meet sufficient methodological standards, such as insufficient statistical power or critical deviations from the original methodology (Stage 1 Primary Criteria #1 and #2). No fixed level of statistical power is required, but will be assessed on an individual basis before results are known. Authors may also be required to revise their Stage 1 submission to add necessary detail to the reported methodology or to alter the analyses. The description of methods should be sufficiently elaborate to provide an internally complete account of all employed methods, without the reader needing to examine the specific methodological details of the original study.

If the manuscript is reviewed favourably according to the Stage 1 Primary Criteria then it will be awarded in principle acceptance (IPA). Authors are then invited to submit a Stage 2 manuscript that includes the content of the approved Stage 1 manuscript plus the results and discussion.

3.3.  At Stage 2 the manuscript is returned to the original Stage 1 reviewers who assess the content according to the following Primary and Secondary criteria:

Stage 2 Primary Criterion #1: Whether the introduction and methods are the same as the approved Stage1 submission (required). 

Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data. 

Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.

Of the above Stage 2 criteria, editorial decisions will be based solely on the Primary Criteria. Comments  regarding Secondary Criterion #1 will not influence the decision to publish but could require additional caveats by the authors (in the discussion) or may be noted by the editors in an accompanying cautionary note.

Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.

4.     Registered Reports track

4.1.  This track is similar to the results-blind process but intended for authors who have yet to attempt a replication of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or any other journal approved by the editors. Authors should initially submit a partial manuscript through the Registered Reports article type containing the abstract, introduction and method sections, written in the future tense. In all other respects, the options and requirements are the same as for the results-blind track.

4.2.  The Stage 1 and Stage 2 submissions will assessed according to the same Primary and Secondary Criteria as for the results-blind track (see 3.2 and 3.3 above)

4.3.  The final Stage 2 paper will be published as a Registered Report.