The stars are not aligned when it comes to online reviews of nursing homes.
A new study by the USC Leonard Davis School of Gerontology shows that Yelp reviewers give nursing homes significantly less favorable ratings than those found on the federal website, Nursing Home Compare, run by the Centers for Medicare and Medicaid Services (CMS).
“Consumers view the quality of nursing homes as poorer than the CMS is reporting it to be,” said Anna Rahman, a co-author of the report and an assistant research professor of gerontology at the USC Leonard Davis School.
Both the Nursing Home Compare website and the popular online business ratings site, Yelp, feature five-star rating systems to help consumers evaluate nursing homes
The study published in the journal BMJ Quality and Safety on Nov. 13 showed that a nursing home’s Yelp rating differed from the Nursing Home Compare rating more than 75 percent of the time, and it was significantly lower in three of four categories reviewed.
Rahman said both ratings systems have weaknesses and strengths.
“Many Americans are now familiar with how Yelp works and evaluate Yelp reviews with a critical eye. They know not to believe everything they read on the site,” Rahman said.
By contrast, consumers have reported difficulty with understanding Nursing Home Compare ratings, she added. This is problematic because prior research indicates that some nursing homes listed on the federal website are gaming their ratings.
“Regrettably, there is no easy or highly intuitive way to spot fake ratings on NHC [Nursing Home Compare],” Rahman said.
“However, we cannot say that Yelp is a better resource than Nursing Home Compare,” she added. “Our best recommendation is that consumers consult both resources, with the caveat that they must understand how each source generates its ratings.”
Different inputs
Nursing Home Compare rolled out its five-star rating system in 2009 using data from nursing home inspections, staffing reports and resident assessments. The overall rating combines scores from three areas: health inspections, staffing levels and quality measures, like changes in physical, functional, psychosocial and cognitive well-being. Health inspections was the only category that rated significantly higher when compared to the Yelp star ratings.
Founded in 2004, Yelp boasts 145 million first-person word-of-mouth reviews of business and service providers. These reviews generally describe a consumer’s personal experience with the provider. Previous research in hospitals shows that Yelp reviewers tend to focus on subjective experiences of health care, such as a reviewer’s personal assessment of staff attitudes, the physical setting or the cost of care.
“Choosing a nursing home for oneself or a loved one is a complex, challenging, often emotionally charged task,” said the study’s co-author, Susan Enguidanos, associate professor of gerontology at the USC Leonard Davis School. “Our study results highlight the need for further research to improve and strengthen the ratings systems used by nursing home consumers.”
Gaming concerns
Both Nursing Home Compare and Yelp have faced charges of artificially manipulated ratings. Previous research estimated that faked reviews and ratings on Yelp range from 16 percent to 40 percent. However, the USC researchers say that since gamed Yelp ratings may encompass either or both extremes of the five-star rating scale, this bi-directional practice, coupled with increasingly savvy consumers and Yelp’s efforts to weed out fake reviews, likely cancels out their ability to impact ratings.
Recent reviews of the Nursing Home Compare system concluded that the rating system provides a financial incentive for gaming, and that the practice of nursing homes reporting purposefully inflated data continues, despite modifications designed to discourage it. These findings may shed light on why the Nursing Home Compare ratings are mostly higher than the Yelp ones, according to the study authors.
“If California nursing homes gamed some of the data they reported to CMS, that could explain why Yelp ratings in our study were significantly lower than the nursing home ratings for quality measures and staffing, which are derived from data self-reported by nursing homes,” said Rahman.
Future work
The authors note that Yelp recently partnered with the New York-based news non-profit ProPublica to report government data about nursing home fines, serious deficiencies and payment suspensions on its review pages. Future studies will aim to analyze the content of Yelp reviews and compare the themes and topics consumers identify to those reported by CMS.
The Ratings Game: An Analysis of Nursing Home Compare and Yelp Ratings was published Nov. 13 at BMJ Quality and Safety. USC Leonard Davis School researchers compared Nursing Home Compare and Yelp ratings for 675 California nursing homes with a minimum of five Yelp reviews between September and November 2016.
