Shots - Health News
2:23 pm
Thu March 13, 2014

Google's Flu Tracker Suffers From Sniffles

Originally published on Fri March 14, 2014 9:27 am

If you want to know what's up with the flu at the moment, you have a few choices: You can get the latest information at Google Flu Trends. Or you can get the official word from the Centers for Disease Control and Prevention, which is based on data that's by now a couple of weeks old.

But a report in the journal Science finds that quicker isn't necessarily better.

This story started in 2008, when scientists at Google realized that they could measure flu activity by tracking when people searched for flu-related terms. They created a handy site called Google Flu Trends. And it works pretty well — some of the time. But don't bank on it.

"It missed by a huge amount last year and actually, it turns out, it's been missing by a fair amount for several years," says David Lazer, a professor of political science and computer science at Northeastern University. He says Google Flu Trend whiffed during the 2013 flu season.

"It's like bases loaded and bottom of the ninth, striking out on three pitches. They predicted twice as many flu cases as the CDC later said there were."

Lazer has written a critique of Google Flu Trends in the latest issue of Science. He finds that the data collected painstakingly from around the country and forwarded to the CDC is still much better, even with the time lag.

"You could just have used old CDC data — two or three weeks old — and have projected forward, and done a better job than Google Flu Trends," he says.

So does this make Google's approach worthless?

"Not at all," he says. "Not at all. I think, actually, the core idea is a terrific one."

Lazar thinks Google could improve its system with the help of outside scientists if it were less secretive about what exactly it's doing to get its results.

But he sees some inherent problems, as well. Google is always refining its search methods, which is good for people doing Google searches, but not so good for analyzing that data — consistency is important in science.

"If we know ahead of time [that] flu is going to really peak in the next few weeks, we need to start getting additional resources to help manage all these patients who are going to be coming in," says Andrea Dugas, a professor at the Johns Hopkins University School of Medicine and an emergency room doctor.

During past outbreaks, Hopkins has actually opened up new areas of the hospital to care for flu patients. Dr. Dugas and her colleagues have been working to improve those predictions, looking at various methods to do that.

"The most accurate of those models was the one that looked at the confirmed flu cases," she says. "That one was the most accurate in predicting what was going to happen in the following week."

When they added in the Google Flu Trends data, their prediction improved, she says, but not by a lot.

"Adding that in really helped refine the model and give us a better prediction," she says, "but the main driver for that was the number of flu cases."

She and Dr. Richard Rothman at Hopkins have developed an app called Flucast to help hospitals predict the ebb and flow of flu.

Another free app, called Flu Near You, also maps out flu trends, based on reports that individuals submit through the app.

Dr. Dugas says Google's approach is no substitute for lab tests, hospital reports and on-the-ground data. And it's important to note that it's not tracking actual flu caused by influenza viruses — rather, it's identifying common symptoms like fever, cough and sore throat.

Google didn't provide a scientist to comment for this story, but a Google report notes one shortcoming: They sometimes get a flood of searches simply when there's a lot of flu in the news. So Google updated the flu tracker system last fall to help reduce the errors that result from that.

Copyright 2014 NPR. To see more, visit http://www.npr.org/.

Transcript

DAVID GREENE, HOST:

OK, raise your hand if you've been on the Google Flu Trends site today. OK, raise your hand if you even knew the Google Flu Trends site existed. Well, it does. And if you go to the site today, you'll see that flu season seems to be winding down. There's a somewhat less rosy picture from the Centers for Disease Control and Prevention website though. That's where the official flu information is reported, though it's a couple of weeks old.

Google started its flu site in 2008 to provide more up-to-date information. But a report in Science magazine finds that quicker isn't necessarily better.

NPR's Richard Harris reports.

RICHARD HARRIS, BYLINE: A few years ago, scientists at Google realized that they could measure flu activity by tracking when people searched for flu-related terms. They created a handy site called Google Flu Trends. And it works pretty well, some of the time, but don't bank on it.

DAVID LAZER: It missed by a huge amount last year. And actually, as it turns out, it's been missing by a fair amount for several years.

HARRIS: David Lazer is a professor of political science and computer science at Northeastern University. He says Google Flu Trend whiffed during the 2013 flu season.

LAZER: This is like bases loaded and bottom of the 9th and, you know, striking out on three pitches. They predicted twice as many flu cases as the CDC later said there were.

HARRIS: Lazer has written a critique of Google Flu Trends in the latest Science magazine. He finds that the data collected painstakingly from around the country and forwarded along to the CDC, is still much better - even with the time lag.

LAZER: You could just have used old CDC data, let's say two or three weeks old, and have projected forward and done a better job than Google Flu Trends.

HARRIS: Is this a useless exercise then?

LAZER: Not at all. Not at all. I think actually the core idea is a terrific one.

HARRIS: Lazer thinks Google could improve its system with the help of outside scientists, if it were less secretive about what exactly what it's doing to get its results. But he sees some inherent issues, as well. Google is always refining its search methods, which is good for people doing Google searches, but not so good for analyzing that data - consistency is important in science.

Andrea Dugas, who's an emergency room physician and a professor at the Johns Hopkins medical school, says Google Flu Trends is going after an important problem.

ANDREA DUGAS: If we knew ahead of time, OK, flu is going to, you know, really peak in the next few weeks, we need to start getting additional resources to help manage all these patients that are going to be coming in.

HARRIS: During past outbreaks, Hopkins has actually opened up new areas of the hospital to care for flu patients. Dr. Dugas and her colleagues have been working to improve those predictions, looking at various different methods to do that.

DUGAS: The most accurate of those models was the one that looked at the confirmed flu cases. That one was the most accurate in predicting what was going to happen in the following week.

HARRIS: When they added in the Google Flu Trends data, their prediction got slightly better.

DUGAS: Adding that in really kind of helped refine the model and give us a better prediction. But the main driver for that was the number of flu cases.

HARRIS: She and colleagues developed an app for hospitals to use, to predict the ebb and flow of flu. Dr. Dugas says Google's approach is no substitute for lab tests, hospital reports and on-the-ground data. And it's important to note that it's not tracking actual flu, just searches of common symptoms like fever, cough and sore throat.

Google didn't provide a scientist to comment for this story. But a Google report notes that part of the issue was that they sometimes get a flood of searches simply when there's a lot of flu in the news. So they updated the flu tracker system this past fall to help reduce the errors that result from that.

Richard Harris, NPR News.

(SOUNDBITE OF MUSIC)

GREENE: This is NPR News.

(SOUNDBITE OF MUSIC) Transcript provided by NPR, Copyright NPR.