The central problem: flawed ME/CFS treatment studies tell us nothing useful

Edwards highlighted the central problem, that CBT and graded exercise studies have relied on biased methods that produce unreliable results.

Most clinical trials of drugs are run blinded, which means that patients don’t know if they are getting the active drug or are receiving a dummy pill in the “control” group.

Success is defined not as the improvement in the drug group alone, but whether or not people in the drug group improve by substantially more than those in the placebo group.

Such blinding is usually required in biomedical treatment studies because it is well known that if patients are asked how they feel, their answers can be biased because they expect to feel better or they want to please researchers. This is known as expectation bias and blinding is there to protect against this.

Blinding is much less of an issue when a study uses objective outcomes, such as changes in blood sugar levels or exercise capacity. However, for subjective outcomes, such as patients scoring their own fatigue levels, expectation bias is a major problem.

Edwards told the committee that bias “is an aspect of human nature that affects all types of scientific experiment, whether in the lab or in the clinic. If outcome measures are truly objective blinding is not necessary, and vice versa. For this reason, unblinded trials with subjective outcomes are specifically considered unreliable.”

The PACE trial as well as all the other psychosocial treatment trials in ME/CFS were unblinded, and so problems of bias, said Edwards, “make more or less all the trials to date unsuitable as a basis for treatment recommendation.”

But does blinding really matter for ME/CFS?

Despite the compelling case for bias undermining trials of CBT and graded exercise, the PACE trial authors and their supporters have tried to justify the continued use of flawed methods.

They have argued that placebo effects in ME/CFS are likely to be small and short-lived. More generally, Professor Sir Simon Wessely states in his “how to” guidebook for running clinical trials in psychiatry that blinding is nice – but not necessary. Wessely has even called the unblinded PACE trial “a thing of beauty”.

Edwards pointed to the clear evidence from rituximab studies that blinding is as essential for ME/CFS studies as it is for any other disease.

Following a promising pilot study of rituximab for ME/CFS, the researchers looked to find the most effective dose of the drug to use.

All the patients in the dosing study knew they were on rituximab. There was also no control group, but all the same, the study showed major gains for patients. Things were looking good.

But a subsequent blinded study by the same team showed that rituximab had no real effect. While those getting the drug improved, just as in the dosing study, a similar, high proportion of those getting the placebo also improved. This result underscores the problem of expectation bias and shows why blinding is so important in ME/CFS studies – despite the claims of PACE authors.

“Unblinded trials like PACE provide us with no useful information”

Edwards went on to point out that drug companies spend hundreds of millions of dollars a year to ensure blinding in trials, and they wouldn’t do that if it wasn’t necessary to meet the high standards needed for drug approval. Whatever Wessely might claim, Edwards said, blinding “is necessary in order to get reliable results”.

He concluded that “Unblinded trials like PACE provide us with no useful information because treatment with no specific beneficial action [such as the placebo in the rituximab trial] can give the same result.”

The psychologist Brian Hughes, who has written about the crisis in psychology research, paraphrased Edwards more bluntly: “bad evidence is no evidence at all”.

Edwards argued that there are two reasons why bias is an even bigger issue in trials of psychosocial therapy than in drug trials.

First, patients are told that the treatments work, increasing the level of expectation bias. Second, patients are encouraged to take a positive attitude to problems, playing down symptoms. This is also likely to lead to patients rating their symptoms as less severe than they really are.

The blinding conundrum – no free pass for psychosocial research

Edwards argued throughout his testimony that the high standards required for drug trials should apply equally to psychosocial treatment trials.

Of course, it simply isn’t possible to conceal from a patient that they are getting CBT or graded exercise. That blinding is not possible does not make the problem go away. Instead, says Edwards, it is down to the researchers to solve this problem in order to get reliable results.

He added, “Unreliable results cannot be seen as reliable just because it is difficult to get more reliable ones.”

Edwards pointed out that it is possible to design a system that ensures that key subjective outcomes are supported by objective ones. This has been done in his field of rheumatoid arthritis.

“Unreliable results cannot be seen as reliable just because it is difficult to get more reliable ones.”

Although there was no such system in place in any ME/CFS psychosocial trials, PACE did measure some objective outcomes. The subjective gains in self-reported physical function and fatigue were not supported by corresponding gains in the objective outcomes. This means, said Edwards, that the best available evidence is that CBT and graded exercise do not work.

Unethical to continue using CBT and graded exercise

Edwards pointed to three ethical concerns with using these treatments.

1. Unsupported theories

Both CBT and graded exercise, argued Edwards, appear to be delivered in a framework that assumes patients have unhelpful beliefs about their capacity for exercise. Edwards said that he had seen no evidence that this view was “any more than popular prejudice”.

The theory that exercise is a route to recovery for people with ME/CFS is equally problematic. Maximal exercise tests, said Edwards, provide at least some evidence that patients’ physiological response to exercise is different in ME/CFS.

2. Evidence of harm

Edwards said that from a theoretical perspective, increasing activity is as likely to harm as it is to help.

Which makes it all the stranger that despite being willing to rely on subjective self-reports of improvements, those backing psychosocial treatments have been unwilling to consider literally thousands of self-reports from patients of harm. Edwards cites Tom Kindlon’s paper exposing the widespread reporting of harms by patients.

Edwards said, “There are good reasons to think psychosocial therapy and exercise therapy can do harm, particularly if they involve misrepresentation of knowledge about an illness.”

3. Claims the treatments work

As Edwards chronicled in his evidence, the treatments don’t work.

Given the problems with the claims made for CBT and graded exercise, Edwards said, it is unethical to tell patients the treatments are based on sound theory, unethical to say they are safe and unethical to claim they work.

Quality control: the “broken field” and the unsayable criticism

That CBT and graded exercise continue to be used and recommended despite the evidence shows, said Edwards, that quality control has broken down.

It became clear then that the existing approach of using unblinded studies with subjective outcomes couldn’t produce useful results. So Edwards was surprised to discover years later that researchers of psychosocial therapies for ME/CFS were still pursuing such a flawed approach.

“the disciplines involved need to take a long hard look at themselves.”

Research quality is supposed to be policed by peer review, where fellow researchers check the standards of studies submitted for funding or publication.

This requires that the peer reviewers understand the problems and reject work that is flawed. It appears that they don’t, as Edwards illustrated with three examples.

1. The most egregious issue with the PACE trial was the authors moving the goalposts for what counted as recovery. They abandoned their original, published definition, replacing it with a much weaker one after they had seen the study’s results.

The authors justified the change by saying that the revised, much more positive results were more consistent with previous studies and their clinical experience. Edwards pointed out that the authors “do not seem to realise that outcome measures need to be predefined in order to avoid exactly this sort of interference from expectation bias”.

2. Cochrane reviews systematically evaluate the evidence on a particular topic. These reviews are generally highly regarded but Edwards highlighted problems with Cochrane work on ME/CFS.

The most recent Cochrane review of CBT for ME/CFS comes to a guarded conclusion about short-term benefit and emphasises the need for more studies. Edwards points to problems that suggest that even these conclusions are overoptimistic.

A more recent review on graded exercise, which concluded it was a good thing, has been heavily criticised. Even Cochrane itself has now expressed reservations about the review.

Both reviews failed to address the problem of bias in unblinded studies. And Edwards highlighted that both reviews were authored by junior colleagues of a PACE trial researcher. This is not the independence that is needed.

3. Edwards discovered first-hand problems with peer review in the field when he submitted a paper that mentioned the problem with subjective outcomes in unblinded trials. Peer reviewers wanted him to drop that point, not because the critique was wrong, “but because it would cast doubt on almost all treatment studies in clinical psychology”.

“Something is badly wrong”, said Edwards, “the disciplines involved need to take a long hard look at themselves.”

Why CBT and graded exercise have to go

Edwards concluded by telling the NICE guidelines committee he believed NICE should not recommend either CBT or graded exercise for ME/CFS. He said this applies to both adults and children. The evidence for the therapies is unreliable because of bias and there are significant ethical problems with their continued use.

Professor Jonathan Edwards’ expert testimony is a brilliant and brutal demolition of the case for CBT and graded exercise as treatments for ME/CFS. While he drew on the evidence of many others, his testimony is perhaps the most impressive single critique of flaws in the evidence used to justify these treatments. It deserves to get a wider audience.

Follow blog by email »

My blog focuses on what seem to me to be the most important points of Edwards’ testimony. You can read his full submission [PDF], starting on Page 5.

Researchers whose work pointing out flaws in ME/CFS research was referenced by Professor Edwards

Lead authors: Tom Kindlon, Dr Carolyn Wilshire, Mark Vink, Alexandra Vink-Niese, Dr Keith Geraghty, Professor Brian Hughes. Co-authors: Robert Courtney (RIP), Alem Matthees, David Tuller, Professor Bruce Levin. Links are to work referenced by Edwards but not included in my blog.