News: Publicly reported quality rating systems need improvement, often conflict, study finds

Publicly reported hospital quality ratings frequently offer conflicting results, which may be misleading to stakeholders and patients, according to a new analysis published in the New England Journal of Medicine Catalyst.

The researchers evaluated four major public reporting sites on their various strengths and weaknesses. Then, they gave them a letter grade; none of the sites received an A or an F. Here’s how each faired:

• U.S. News & World Report: B
• CMS Star Ratings: C
• Leapfrog Group: C-
• Healthgrades: D+

Each rating system, according to the researchers, had “unique weaknesses that led to misclassification of hospital performance.” These  ranged from inclusion of flawed measures, use of unvalidated proprietary data, and methodological decisions.

There were also several issues that limited all the examined rating systems:

• Limited data and measures
• Lack of robust data audits
• Composite measure development
• Measuring diverse hospital types together
• Lack of formal peer review of their methods

These inconsistencies and inadequacies, according to the researchers, mean that the rating systems should be used cautiously as they often “misclassify hospital performance and mislead” the public.

The researchers also offered several solutions that could help improve the rating systems such as including:

• Better data not based on self-reported data
• Better measures built upon the data pulled directly from the EHR
• Meaningful audits of each rating system to ensure it’s accurate before going public
• External peer reviews, much like the type of review required to publish in an industry publication

“[Until the systems are improved,] these rating systems should be interpreted very cautiously,” the researchers wrote. “Most seem likely to misclassify hospital performance and may mislead patients, referring doctors, payers, purchasers, and hospitals themselves.”

Editor’s note: To read the full analysis, click here. To read about a recent study saying that 70% of patients say online reviews are crucial to selecting providers, click here.