Thursday, 16 March 2017

Should all registered clinical trials be published as Registered Reports?

Last night I asked you brilliant folks to give me your strongest counterarguments to the following proposition: that all registered clinical trials should be published in journals only if submitted as Registered Reports (RRs).

It was a really interesting discussion, and thanks to everyone who engaged. Here are the reasons that were put forward. 

My tl;dr verdict: I’m still waiting for a good reason!

1. RRs require presenting more methodological detail than standard clinical trial registration and so expose authors to a potential competitive disadvantage.
This isn’t really a scientific objection (or least, it’s a very weak scientific objection) but I understand the strategic argument. My response is that if all registered trials have to published as RRs then everyone faces the same disadvantage, so there is no relative disadvantage.

2. Clinical trial registration is sufficient for controlling bias.
It’s not. Around 16-66% of clinical trials never report results, ~14% are registered after data collection is complete, and somewhere between 30-85% engage in hidden outcome switching. With depressing statistics like this, how can standard trial registration be seen as anything close to sufficient?

3. OK, clinical trial registration used to be insufficient but it’s sufficient now because clinicaltrials.gov requires authors to specify a primary outcome measure.
Still nope. The COMPARE project finds that authors routinely engage in hidden outcome switching even when primary outcome measures are specified. There is no logical reason why requiring something that already fails to prevent hidden outcome switching should prevent hidden outcome switching.

4. Ok fine, but a signed declaration at submission that outcomes haven’t been switched would solve hidden outcome switching.
It would probably have some effect, but then it remains easy to specify an outcome sufficiently vaguely to enable one of several variables to be cherry picked as the primary outcome measure, and so allow researchers to tick this box even when they switched. And even if this measure did reduce outcome switching, it would not reduce publication bias. RRs reduce both hidden outcome switching and publication bias. So why should any kind of declaration be preferable to all registered clinical trials being published as RRs?

5. Small companies often live or die by the results of trials. The RR model presents a risk to their livelihoods if they have to publicly admit that an intervention failed to work.
The model suggested here applies only to registered trials. If companies want to do their own internal unregistered trials and choose what to publish (where they can) based on the results, that’s up to them. The argument here is that the price of attaining credibility within the pages of a reputable peer-reviewed journal should be to register the trial as a RR.

6. RRs involve one paper arising per protocol. But a single protocol may need to produce multiple papers addressing different questions. This is also important to support the careers of early career researchers.
This sounds to me like an argument for salami slicing in the interests of careerism. But I accept that in the reality of academia, careers matter. My initial reaction is that if the research question and method are complementary enough to go in the same protocol, why aren’t the results complementary enough to go in the same paper? The easy solution to this is to separate protocols that address different questions into different RRs. That way there are as many papers to publish as there are separate research questions.

7. The RR model doesn’t force authors to publish their results. Therefore there is no guarantee that it will prevent publication bias.
This is the strongest objection so far, but even so it is virtually guaranteed to be less of a problem for RRs than under the status quo. Authors of RRs are indeed free to withdraw their papers after the protocol is provisionally accepted, but doing so triggers the publication of at least part of the registered protocol plus a reason for the withdrawal. So what is an author going to say, that they withdrew their peer-reviewed RR because the trial outcomes were negative? I suspect the research community (including the reviewers who invested time in assessing the protocol) would take a dim view on such a strategy, and it is probably for this reason that there has yet to be a single case of a withdrawn RR at any journal. In any case, if this were considered to be a serious risk, it would be straightforward to strengthen the RR model for clinical trials so that it requires authors to publish the results. If every journal published registered clinical trials only as RRs, and all RRs bore this mandate, virtually all registered trials would be published.

So that’s seven reasons, none of which I think are particularly strong.

Got anything stronger?

2 comments:

  1. Interesting suggestion! As a member of the general public and a former student i am amazed that there seem to me to be such low standards in science. I think performing clinical trials as Registered Reports is a great idea to adhere to some higher standards and improve science, and i wonder why the majority of scientists don't come together and make this happen.

    In trying to think about why this does not seem to be the case at this point in time, i wonder 2 things:

    1) How do you think the general public views the possibility of clinical trials being performed as Registered Reports?

    2) If the general public has a different view on this issue than researchers, why could this the case in your view?

    ReplyDelete
  2. I found one example where the same trial was registered on two different CT registries, but the primary and secondary outcomes listed on those registries were not identical. It didn't look deliberate in this case - the trial was sponsored by a UK university and overall their registry entries are a terrible mess - but it's a risk factor to bear in mind. For details, see the UK universities study at www.TranspariMED.org

    ReplyDelete