The Patient Safety Indicators (PSIs) were evaluated by the project team using empirical analyses to explore the frequency and variation of the indicators, the potential bias, based on limited risk adjustment, and the relationship between indicators. The data sources used in the empirical analyses were the 1997 Florida State Inpatient Database (SID) for initial testing and development and the 1997 Healthcare Cost and Utilization Project (HCUP) State Inpatient Database for 19 States for the final empirical analyses.
All potential indicators were examined empirically by developing and conducting statistical tests for precision, bias, and relatedness of indicators. Three different estimates of hospital performance were calculated for each indicator:
- The raw indicator rate was calculated using the number of adverse events in the numerator divided by the number of discharges in the population at risk by hospital.
- The raw indicator was adjusted to account for differences among hospitals in age, gender, modified Diagnosis-Related Group (DRG), and comorbidities.
- Multivariate signal extraction methods were applied to adjust for reliability by estimating the amount of "noise" (i.e., variation due to random error) relative to the amount of "signal" (i.e., systematic variation in hospital performance or reliability) for each indicator.
The project team constructed a set of statistical tests to examine the precision, bias, and relatedness of indicators for all accepted Provider-level Indicators, and precision and bias for all accepted Area-level Indicators. It should be noted that rates based on fewer than 30 cases in the numerator or the denominator are not reported.
The project team conducted a structured review of each indicator to evaluate the face validity (from a clinical perspective) of the indicators. The methodology for the structured review was adapted from the RAND/UCLA Appropriateness Method and consisted of an initial independent assessment of each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists, followed by a final independent assessment by clinician panelists using the same questionnaire. The review sought to establish consensual validity, which "extends face validity from one expert to a panel of experts who examine and rate the appropriateness of each item..." The panel process served to refine definitions of some indicators, add new measures, and dismiss indicators with major concerns from further consideration.
Refer to the original measure documentation for additional details.
