Resources

Measure Validity

Using the NQMC Template of Measure Attributes to Assess the Validity of Measures

Some measures produce less valid results than other measures. The validity of results depends upon how the measure is built and whether it is built to address the purpose chosen by the user. The NQMC Complete Measure Summary captures key building blocks that you can use to assess the validity of a measure for your purpose.

NQMC contains measures grouped into 7 measure domains.

Measures of access, process, outcome, and patient experience assess the quality of care provided by health care professionals and organizations.

Measures of structure assess the capacity of health care professionals and organizations to provide high quality of care.

Note: These 5 domains include measures of the quality of care.

Two other domains in NQMC include measures that are used to monitor trends in use of services and population health; but these other domains are not direct measures of the quality of clinical care.

You should consider how well your intended use of the measure matches the developer's intended use. The use intended by the measure developer is captured in the "State of Use of the Measure" section of the NQMC Complete Measure Summary, especially the "Current Use" field, and also in the "Evidence Supporting the Need for the Measure" section.

Five key questions may help you to decide about a measure's validity as a measure of quality. The NQMC Complete Measure Summary has specific fields that provide an answer to each question.

Question 1. How strong is the scientific evidence supporting the validity of this measure as a quality measure?

The key field from the NQMC Complete Measure Summary for this question is "Evidence Supporting the Criterion of Quality" which applies to the domains of access, outcome, patient experience, process and structure. Comparable fields address evidence supporting the value of monitoring of population health and use of services.

Example: For the access measure, "Mental Health Intensive Care Management," the field, "Evidence Supporting the Criterion of Quality" characterizes the evidence as consisting of "A clinical practice guideline or other peer-reviewed synthesis of the clinical evidence," as well as, "One or more research studies published in a National Library of Medicine (NLM) indexed, peer-reviewed journal."

Question 2. Are all individuals in the denominator equally eligible for inclusion in the numerator?

A valid measure of quality of care should exclude individuals that should not receive the indicated care or are not at risk for the outcome. The key field addressing this question is "Relationship of Denominator to Numerator."

Example: For a maternity care measure that assesses the percent of patients who received substance abuse treatment services, this field would reveal that all cases in the denominator are not equally eligible to be included in the numerator. Not all mothers delivering newborns need or should receive substance abuse services, only those who have substance abuse problems. For instance, imagine two health plans, A and B: in Plan A, 10% of mothers are substance abusers and 1% are treated. In Plan A then, there is gross under use - only 10% of the mothers in need get treatment. In Plan B, only 1% of mothers are substance abusers and those 1% are treated; this is perfect care because 100% of the mothers in need get treatment. The measure result shows that Plans A and B have identical scores (both score 1%), even though Plan B clearly provides higher quality care (100% of mothers in need treated). This measure can be used to monitor use of services in the two Plans, but cannot provide a direct measure of the quality of care.

Question 3. Is the measure result under control of those whom the measure evaluates?

The key field for this question is "Measure Results under Control of Health Care Professionals, Organizations, and/or Policymakers." It captures whether the events that are the subject of the measure are somewhat or substantially under control of the targets of measurement.

Example: A measure of asthma prevalence within a Health Plan is population health measure. Clinicians can diagnose asthma, but asthma is primarily caused by genetic and environmental risk factors, not by receiving health care. A user should be wary of using this measure to compare health care providers who care for populations that differ in their risk for developing asthma. A measure developer would use this measure to monitor how many enrollees within a Plan are known to have asthma, not as a direct measure of quality. A direct measure of success in detecting asthma would require two steps, firstly identifying all persons with asthma within the population of enrollees, then checking the percentage of these persons that were known by the Plan to have asthma.

Question 4. How well do the measure specifications capture the event that is the subject of the measure?

The key section from the NQMC Complete Measure Summary for this question is "Data Collection for the Measure." Within this section, the fields, "Numerator Inclusions/Exclusions" and "Denominator Inclusions/Exclusions" show the details of measure construction.

Example: For a measure of that is used to assess hospital admission rates for long-term complications among patients with diabetes mellitus, the fields, "Numerator Inclusions/Exclusions" and "Denominator Inclusions/Exclusions," may describe the use of inpatient administrative data with diagnostic codes for renal, eye, neurological, circulatory and other complications of diabetes. This would enable you to review, for example, whether codes for appropriate complications are included in the measure.

Question 5. Does the measure provide for fair comparisons of the performance of providers, facilities, health plans, or geographic areas?

For some measures, it may be important to account for differences in the characteristics of individuals that receive care from different providers, facilities, or health plans, or that live in different geographic areas. This may be done through statistical adjustment or stratification of the sampled population for the measure. The key fields for this question are "Allowance for Patient Factors" and "Description of Allowance for Patient Factors." These fields reveal whether the measure includes allowance for patient factors and describe what those factors are. Allowance for patient factors is critical for measures of the outcomes of care because, in addition to the treatments given, demographic and clinical factors may influence the measure result. Allowing for patient factors may be a concern for other types of measures also.

Example: Results for an outcome measure for mortality rates among acute stroke patients will be largely determined by the patients' underlying health conditions present before the onset of the stroke. The field "Allowance for Patient Factors" may show the methods of risk adjustment used in the measure to enable fair comparison of the quality of care in hospitals.

Measure Validity

Using the NQMC Template of Measure Attributes to Assess the Validity of Measures

Search

Browse

Compare