Self-Diagnosing Online? Study Finds Sites Are Only Accurate About Half Of The Time

“Looking at whether these tools are good enough to replace the doctor is the wrong debate,” said Jason Maude, co-founder of Isabel. (Robin Lubbock/WBUR)

“Looking at whether these tools are good enough to replace the doctor is the wrong debate,” said Jason Maude, co-founder of Isabel. (Robin Lubbock/WBUR)

There’s a new warning for those of us who go online to figure out why we have a stomach ache or a nagging cough or occasional chest pain.

Symptom checkers — those tools that let you enter information and then produce a diagnosis — are accurate about half of the time, according to a study out of Harvard Medical School.

How Symptom Checkers Rate

Rate at which each tool got a diagnosis correct in the first three suggestions:

Best Performing:
Symcat – 75 percent
Isabel – 69 percent
AskMD – 68 percent

Worst Performing:
BetterMedicine – 29 percent
Earlydoc – 33 percent
Symptomate and Esagil – 34 percent

Source: Harvard Medical School study (full table page 11)

Looking at 23 websites, the Harvard study found that a third listed the correct diagnosis as the first option for patients. Half the sites had the right diagnosis among their top three results, and 58 percent listed it in their top 20 suggestions.

“Users of these tools should be aware that their performance is not perfect by any means, there’s often inaccuracies or errors,” said Dr. Ateev Mehrotra, the study’s lead author.

At the Mayo Clinic, Dr. John Wilkinson said, “We’re always trying to improve, but if most of the time the diagnosis is included in the list of possibilities, that’s all we’re attempting to do.”

Wilkinson, an editor of Mayo’s symptom checker, said patients should not expect it to deliver the correct diagnosis.

“It’s designed to be a starting point,” Wilkinson said, one that will direct patients to the best articles and help them “be better equipped to have a conversation with their doctor or a nurse triage line or whatever the next step might be.”

Mehrotra says these online tools are about as accurate as the call-in lines offered by many insurers and physician groups. “[They are] better than just a random Internet search,” Mehrotra said, “but they are not a replacement for going to the doctor and getting a full evaluation and diagnosis. They are simply providing some information about what might be going on with you.”

Jason Maude, co-founder of Isabel, which was a top-ranked site in the study, urges patients to use his tool as a second opinion, after they’ve seen a doctor.

“Looking at whether these tools are good enough to replace the doctor is the wrong debate,” Maude said.

A well-informed patient, Maude says, can work with their doctor to reach the right diagnosis from among dozens, if not hundreds, that might apply, then decide on the best plan for care.

“I don’t know of many other industries where there’s such a big difference between the professional and the consumer,” Maude said. In medicine, most patients “often want to rely entirely on what the doctor says, but you can’t really do that.”

Isabel may have better results, Maude said, because it lets patients type in their own description of symptoms. They might describe a “tummy ache” or “stomach cramps” rather than the more clinical choice of “abdominal pain” used by many online symptom checker tools. And Isabel asks just two or three questions before patients describe their problem, as compared to sites that ask patients to click through 20 questions — steps Maude said may discourage use.

The Harvard study used 45 patient cases, ones used to teach medical students how to reach a diagnosis. Researchers put the case study patient symptoms into the online tools and checked the results.

Mehrotra said the symptom checkers need to warn patients that the results are just advice, “but are not gospel, not perfect.”

The study also looked at how often these online tools recommended the right next step for patients. The sites do a good job of assessing the need for urgent or emergency care, but don’t do so well in cases where a patient is better off waiting a day or two or caring for themselves:

Correct next step provided by symptom checkers by type of illness (percentage)

– Illness requiring care right away: 80%

– Illness where care in next day or two is OK: 55%

– Illness where staying at home is fine: 33%

Mehrotra says patients used symptom checkers more than 100 million times last year, a fact that may stun some physicians.

“It’s been very surprising to me that while most doctors know patients are going to the Internet to search for medical advice, in terms of these symptom checkers, I’ve been surprised that few of my colleagues even knew they existed,” he said.

— Here’s a video recap of the study from the British Medical Journal:

Please follow our community rules when engaging in comment discussion on this site.