Steps for Assuring Rigor and Adequate Patient Representation When Using Patient-reported Outcome Performance Measures

On behalf of the Board of Directors of the International Society for Quality of Life Research (ISOQOL)

Several recent initiatives have identified methodological standards for developing and implementing patient-reported outcome (PRO) performance measures. (1, 2, 3)

These measures, which can be referred to as PRO-PMs, are defined as a PRO embedded within a set of specifications to assure standardization and comparability of performance results. One example of a PRO is the concept of depression; a related example of a PRO measure is the Patient Health Questionnaire (PHQ-9) depression module. An example of a PRO-PM from the PHQ-9 is the percentage of patients with a diagnosis of major depression or dysthymia and initial PHQ-9 score >9 who demonstrate remission at 6 months with a follow-up PHQ-9 score <5 (National Quality Forum [NQF] endorsed measure #0711). (4)

These standards largely address the need for clinically relevant and patient-centered measurement of symptoms, functional status, and health-related quality of life. Assessing patient experiences with care delivery (e.g., satisfaction with care) is already a well-established discipline and is beyond the scope of this discussion.

In late 2012, the U.S. NQF published a pathway for developers of PRO-PMs to follow that can lead to NQF endorsement. (1) During the same period, the American Medical Association-convened Physician Consortium for Performance Improvement hosted a technical panel that developed a similar series of methodological best practices. The U.S. National Committee for Quality Assurance has been working with PRO methodologists to develop PRO-PMs for collaborative projects with the Centers for Medicare & Medicaid Services and with the Office of the National Coordinator for Health Information Technology. Previously, the King's Fund published an overview of the U.K. PROMs Programme to measure provider performance in the National Health Service. (4) More recently, the American Society of Clinical Oncology has formed a committee to create PRO-PMs for its Quality Oncology Practice Initiative.

These initiatives reflect substantial agreement about the need to establish and harmonize methodological approaches. (5, 6) Members of the International Society for Quality of Life Research (ISOQOL) have been involved in most of these initiatives and worked to assure harmonization.

We have identified several key themes from these initiatives, which can be summarized as recommended steps for organizations that seek to develop and implement PRO-PMs. The ISOQOL Board of Directors has endorsed all of these recommendations:

• Identify a quality performance issue, problem, or knowledge gap in a given population or health care context based on input from multiple stakeholders including consumers, patients, and/or caregivers. Example: little is understood about how well patients' nausea is controlled following chemotherapy.
• Identify outcomes that are meaningful to patients and that are amenable to change or are actionable in the identified population, based on patient engagement activities. Determine whether PRO measurement is the appropriate approach for the identified outcome (for example, the outcome has a subjective component and the target population has the physical and cognitive ability to self-report). Example: nausea prevention and control.
• Identify or develop a PRO measure with adequate psychometric properties for the intended use (i.e., a measure that is understood by and meaningful to patients and that is valid, reliable, responsive, and feasible to implement, including an assessment of patient burden). Example: nausea measures developed by the Multinational Association of Supportive Care in Cancer (MASCC) or the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).
• Pilot test the PRO measure in a real world setting: Assess responsiveness to intervention, actionability, and feasibility; collect information for subsequent risk and case-mix adjustment; refine strategies for implementing the measure and minimizing or otherwise handling missing data; and perform software usability testing (when appropriate).
• Specify the PRO performance measurement criteria and a framework for interpreting and reporting results (e.g., score changes that are meaningful to patients; average change or percentage of patients who improve by a specific amount or meet a benchmark; definition and procedures for identifying outliers; ways to estimate the impact of missing data; units of analysis). Example: the proportion of patients receiving moderately or highly emetogenic chemotherapy who report moderate or severe nausea severity during the week following treatment.
• Evaluate the PRO performance measure against standard NQF endorsement criteria* (i.e., importance; scientific acceptability of measure properties; feasibility; usability and use; and comparison to related and competing measures).
• Continuously evaluate and refine the PRO performance measurement strategy and its impact on practice and outcomes following implementation.

The role of patient engagement is essential through all these steps; it can help to ensure the patient-centeredness of the measures, data collection approach, and reporting strategy. (7, 8) Focus groups or other qualitative methods are important in gaining a diversity of perspectives, particularly with respect to race, ethnicity, culture, education, age, sex, geography, and disease severity. Inadequate representation of a key sub-group in the design and evaluation of the PRO measure may yield quality measures that are not meaningful to sub-groups; for example, a PRO-PM that includes a measure of role functioning may not be valid for populations that do not participate in the work force.

Developers should use both qualitative methods (e.g., cognitive interviewing) and quantitative methods to evaluate the validity, reliability, and feasibility of the PRO measure. Patients, caregivers, or both should be involved in evaluating the implementation plan for the PRO-PM.

Avoidance of missing data from subpopulations necessitates the use of inclusive data collection strategies. Measures should be accessible via several different modes of administration, including the ability for patients to respond verbally to questions (e.g., via automated telephone systems or interviewer-administered questionnaires).

To minimize missing data, investigators should actively monitor compliance with questionnaire completion in near real-time, with backup mechanisms such as automated reminders or human telephone calls. This is best achieved through the use of electronic reporting platforms. Should substantial missing data be observed, analysts should be prepared to conduct sensitivity analyses to evaluate the impact of missing data on study results.

To engage patients successfully requires planning and expertise. Developers of quality measures that require patient reporting are advised to partner with experts in PRO assessment and patient engagement. An extensive body of experience and methods has been developed and refined in other areas of health care evaluation, such as clinical trials and community-based participatory research. (9, 10) Developers of performance measures can capitalize on these experiences.

In summary, good measurement practices lead to better quality data. By employing methodological rigor and engaging the full spectrum of patient populations, information from PRO-PMs will be more meaningful and support patient-centered decision-making by patients, providers, payers, and healthcare systems. (1, 11)

*Available at: www.qualityforum.org/docs/measure_evaluation_criteria.aspx  .

Disclaimer

The views and opinions expressed are those of the author and do not necessarily state or reflect those of the National Quality Measures Clearinghouse™ (NQMC), the Agency for Healthcare Research and Quality (AHRQ), or its contractor ECRI Institute.

Authors/Potential Conflicts of Interest

References

1. National Quality Forum. Patient-reported outcomes (PROs) in performance measurement. 2013 Jan 10. [internet]. [accessed 2013 Jun 22]. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx  .
2. Devlin NJ, Appleby J. Getting the most out of PROMS. London: King's Fund. 2010. Available at: https://www.kingsfund.org.uk/sites/files/kf/Getting-the-most-out-of-PROMS-Nancy-Devlin-John-Appleby-Kings-Fund-March-2010.pdf  .
3. Basch E, Torda P, Adams K. Patient-reported outcomes-based performance measures. JAMA. 2013 Jul 10;310(2):139-40.
4. Minnesota Community Measurement. Minnesota health scores overview. [internet]. [accessed 2013 Jun 13]. Available at: http://www.mnhealthscores.org  .
5. Acaster S, Cimms T, Lloyd A. Development of a methodological standards report. The design and selection of patient-reported outcomes measures (PROMS) for use in patient-centered outcomes research. A report prepared for the Patient-Centered Outcomes Research Institute (PCORI). Submitted to PCORI 2012 Mar 22. Available at: http://www.pcori.org/assets/The-Design-and-Selection-of-Patient-Reported-Outcomes-Measures-for-Use-in-Patient-Centered-Outcomes-Research.pdf .
6. Reeve BB, Wyrwich KW, Wu AW, Velikova G, Terwee CB, Snyder CF, Schwartz C, Revicki DA, Moinpour CM, McLeod LD, Lyons JC, Lenderking WR, Hinds PS, Hays RD, Greenhalgh J, Gershon R, Feeny D, Fayers PM, Cella D, Brundage M, Ahmed S, Aaronson NK, Butt Z; on behalf of the International Society for Quality of Life Research (ISOQOL). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. Epub 2013 Jan 4.
7. Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI). Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA. 2012;307(15):1636-40.
8. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, Tunis SR. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181-194.
9. Brundage M, Blazeby J, Revicki D, Bass B, de Vet H, Duffy H, Efficace F, King M, Lam CL, Moher D, Scott J, Sloan J, Snyder C, Yount S, Calvert M. Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards. Qual Life Res. 2013 Aug;22(6):1161-75.
10. Fairclough DL. Design and analysis of quality of life studies in clinical trials (2nd edition). Boca Raton (FL): CRC Press. 2010.
11. Harrison PL, Pope JE, Coberley CR, Rula EY. Evaluation of the relationship between individual well-being and future health care utilization and cost. Popul Health Manag. 2012 Dec;15(6):325-30.