Matthew Herper and Andrew Joseph write in STAT that two of the world’s leading medical journals on Tuesday expressed concern about potential flaws in the data produced by a small company to draw major conclusions about Covid-19 — that certain heart drugs are safe, and that the malaria drug hydroxychloroquine is not. The latter finding led to the pause of an important study of hydroxychloroquine by the World Health Organization.
“Serious questions have been raised about the reliability of the findings reported in this paper,” Richard Horton, the editor of the Lancet, where the hydroxychloroquine study was published, wrote on Twitter.
Eric Rubin, the editor of the New England Journal of Medicine, which published the study of heart drug safety, struck a similar tone.
“Substantive concerns have been raised about the quality of the information in that database,” Rubin wrote in the so-called expression of concern published by the journal. “We have asked the authors to provide evidence that the data are reliable.”
The concerns, which have built over the past several days on social media, highlight larger issues with using big databases to draw conclusions about medicines, an approach that has been gaining rigor in the era of big data. Experts warn that conducting such studies properly is far more difficult than it appears.
“This is not for the faint of heart,” said Harlan Krumholz, director of the Center for Outcomes Research and Evaluation at Yale New Haven Hospital. “This is not just a matter of dial-a-study when you get access to data. Well-done studies are based on understanding the provenance of the data and making sure what you are doing is reasonable. There is good science to be done with big databases, but there are also major mistakes to be made. The question is: What happened here?”
Both studies in question used data from Surgisphere, a little-known company based in Chicago that claimed in the Lancet study to have data from 671 hospitals on six continents. The Lancet paper found that the malaria drugs chloroquine and hydroxychloroquine, which had been explored as potential therapies for Covid-19, did not correspond with improved outcomes for patients, and were also associated with higher mortality. The paper in the New England Journal of Medicine reported that blood pressure medications were not associated with worse outcomes in patients with Covid-19. The studies share some of the same authors, including Sapan Desai, who runs Surgisphere.
After suspending the clinical trial arm focused on hydroxychloroquine, the WHO said it would review the data generated so far. That portion of the WHO’s multidrug Solidarity Trial remains paused, delaying answers on a drug that has become a political flashpoint even as the evidence on its potential benefits and risks is murky.
But news reports and experts have raised questions about the integrity of the Surgisphere data. The Guardian, for example, reported on discrepancies in data said to come from Australia in the Lancet study. The Lancet study’s authors have corrected errors about the number of participants, but have stood behind their conclusions.
Outside experts, however, have raised broader concerns. In a May 28 letter to the editor of the Lancet and the study’s authors, more than 180 scientists outlined questions about the statistical analysis and a lack of transparency. The study’s authors did not release the full data underlying the study and did not say which countries or hospitals contributed data, the letter says. The outside scientists called on Surgisphere to at least provide details about how it sourced its data and called for an independent validation of the analysis.
On Tuesday, the group of concerned researchers also wrote to the editor of the New England Journal, raising “similar issues” as they had seen in the Lancet paper. One issue, according to the letter: In some countries, “a relatively small number of hospitals are reported to have provided electronic patient record data to Surgisphere, yet these reports describe a remarkably high proportion of all PCR-confirmed cases in the respective countries.” They flagged specific concerns with data purported to come from the United Kingdom and Turkey.
In a statement, the lead author of the two studies, cardiologist Mandeep Mehra of Brigham and Women’s Hospital, said that the research team “initiated independent reviews of the data used in both papers after learning of the concerns that have been raised about the reliability of the database.” Mehra said the decision for the outside reviews came the remaining co-authors of the papers, “independent of Surgisphere.”
“The goal of the independent third-party auditor is to verify the source data and assess the accuracy of the database and the authors’ findings,” Mehra said.
The Lancet and New England Journal on Tuesday separately issued their expressions of concerns.
In their notice, the Lancet’s editors said the independent audit of the data was “expected very shortly” and noted the “serious scientific questions” that had been raised.
Surgisphere did not immediately respond to a request for comment. In a statement on its website after criticisms were raised about the Lancet study, it highlighted “the validity of our database.” It said its registry was based on electronic health records from customers of its machine learning program and data analytics platform, which allow the company to in turn use data from the records to study “real-world, real-time patient encounters.”
“Together, we stand behind the integrity of our studies and our scientific researchers, clinical partners, and data analysts,” the statement said.
The results of the hydroxychloroquine and chloroquine study in particular grabbed attention, as the drugs’ potential as Covid-19 treatments has become politicized. President Trump and his allies have touted their purported benefits, even as no gold-standard clinical trials have yet produced results and observational studies cast doubt on their efficacy. Trump said he was taking hydroxychloroquine to try to protect himself from the coronavirus.
But there have been concerns raised about the potential side effects of the drugs in people with Covid-19; the Food and Drug Administration, for example, has said it should not be used outside of clinical trials or for patients who are not hospitalized, because of the risk it poses to heart health. Trump’s critics used the Lancet study to argue that by promoting the drug, the president had been endangering the public.