Performance Measures for Evaluation and Quality Improvement in the Care of Individuals with a First Episode Psychosis

Measurement-based quality improvement has begun to address priority areas in psychiatry (1). Despite these efforts, the National Committee for Quality Assurance has reported gains in quality of general medical and surgical services in the United States, but not in the quality of mental health services (2). For a patient who will go on to develop schizophrenia, the first episode of psychosis is a pivotal time when the quality of care received can significantly influence his or her future course of illness. In this commentary, we describe the development and pilot testing of performance measures to reduce variability and improve quality of care for this population.

Background

Schizophrenia. Schizophrenia has a lifetime prevalence of about 1%. It typically begins in young adulthood and has a permanent impact, with major effects on the likelihood of marriage, employment and mortality. On a per capita basis it is the single most expensive of the mental disorders and in the World Health Organization assessment of disease impact, it ranks in the top ten of all health disorders. Despite improvements in evidence based pharmacological and psychosocial treatment, many patients do not receive evidence based care (3).

First psychotic episode. Two aspects of the onset of a first psychotic episode carry significant implications for treatment and, in some cases, quality of care.

First, the duration of untreated psychosis, defined as the time between onset of diagnosable symptoms of psychosis and treatment of psychosis, is significant because there is consistent evidence that longer duration of untreated psychosis is associated with poorer outcomes. The onset of schizophrenia can be insidious or sudden. Even after the onset of psychotic symptoms, there are often delays of one to two years before the start of appropriate treatment, with delays being longer in those with an insidious onset. The duration of untreated psychosis can be shortened through a combination of public and gatekeeper education and removing access barriers.

Second, the response to treatment of the first episode is quicker and more complete than the response to subsequent episodes, leading to the implication that prevention of the second episode has both short term and longer term benefits. For example early, appropriate treatment of first episode psychosis can reduce the two year relapse rates from 60% to 30% (4). These reductions in relapse rate can be achieved by providing consistent evidence based care, including psychosocial services, usually through specialized treatment teams.

Specialty services. Over the last 30 years specialized first episode psychosis services have been developed around the world but are not yet a broadly accepted or consistent feature of care in most developed countries. Initially these were research oriented programs, some of which conducted randomized controlled studies which demonstrated that the programs were both effective (5) and cost effective (6). As the evidence has mounted many centres have developed such programs. The core values, goals and standards for first episode services have been articulated in a joint statement of the International Early Psychosis Association and the World Health Organization (7). The International Early Psychosis Association lists 152 self reported programs in 26 countries. This self-reported list is not comprehensive and the programs vary in size and scope. The largest numbers are in Australia, Canada, New Zealand and the United Kingdom, where some national or provincial policies and guidelines have been developed.

The key clinical components for such programs include:

• Public and gatekeeper education
• Easy access
• Pharmacotherapy
• Continuity of care
• Assertive case management
• Family education and support
• Patient education and support
• Integrated addictions treatment
• Supported employment

The impetus for the development of specialty services for individuals with a first psychotic episode is that care is believed to vary widely, leaving the potential for early signs of psychosis going unrecognized, delays in initiation of antipsychotic medications, fragmented rather than continuity of care, and overreliance on pharmacotherapy without accompanying psychosocial interventions that have been shown to improve outcomes. We initiated the development of performance measures that could be employed within the specialty services, to ensure that the care provided was consistent with the goals of these programs.

First episode psychosis services represent a good target for performance measurement development for many reasons. New programs are being funded without required performance measures despite the fact that they represent a significant investment. Although such programs are cost effective if they reduce admissions, failure of effective implementation may result in increased costs. Furthermore while clear policy and practice manuals exist it is not evident that these policies and practices are being adhered to in newly established programs.

Performance Measure Development

The development of new evidence based services for first episode psychosis has provided an opportunity to identify specific evidence based performance measures for such services. In one study a two stage process was used to identify potential measures employing, first, a systematic review of the literature and, second, a Delphi consensus approach to reduce the number to 24 measures rated on one global scale as being essential (8). Subsequently, operational definitions were developed through an iterative process between an expert panel and health information experts (9).

Data sources. The data for calculating the performance measures were obtained by research staff with access to the health facilities systems data bases, health records and research data bases. The source of each performance measure is identified in Table 1.

Pilot testing. Nineteen of these measures (see Table 1) were successfully used to compare two Canadian publicly funded programs that provide mental health care to individuals who experience a first psychotic episode. The sample size required to compare the programs was based on a power analysis using the rate of hospital admission after program entry at two years as the primary outcome measure. The sources of data for each performance measure are indicated in the table along with some comments about the indicators.

The two programs had similar rates on the performance measures (10). Results on the hospitalization measure were similar to results published in randomized controlled studies (11) suggesting that there was accumulating data to establish standards for such programs. The key results are outlined in Tables 2a and 2b.

A number of performance measures were not used because the data were not available, e.g., such as data on cost effectiveness. This has led to attempts to find proxy measures of these concepts. One proxy measure that we have discussed locally is a ratio based upon the average case load per Full Time Equivalent of clinical (non-administrative) staff over the two year hospitalization rate. This has yet to be tested. Another performance measure for which there was no data was evidence of patient involvement in decision making. This was derived from organizations that have standards that require patients to sign care plans. This result suggested a specific quality improvement opportunity for each organization.

Our research next focused on hospitalization as a key performance measure since it is reliable, valid and generally available. The hospitalization rate is the proportion of all patients admitted to hospital within a specified time period, usually one or two years. Hospitalization can be considered a process measure because it is a form of treatment. It is not an outcome such as death, being at work or in education, or clinical relapse which is an increase in symptoms. It can be considered a proxy outcome for clinical relapse because it often results from a relapse. Relapse is hard to reliably assess in clinical settings (12), so hospitalization is used as a proxy. Hospital admission is an important outcome measure since it is universally available (13). It is practically significant because it represents an outcome which is negative for both patients and families and carries significant societal costs (14). Furthermore, it is influenced by a number of important process measures such as medication adherence (15) and family education and support (16). We have developed a risk adjustment model for hospitalization for patients with first episode psychosis that allows for the beginning of comparisons between real world programs (17). To do this, candidate predictor variables for hospital admission were identified through a literature review. An expert panel comprised of an epidemiologist, health services researcher and clinical trial specialist who focused on schizophrenia then selected 11 potential risk adjustment variables through the use of a structured process, the Template for Risk Adjustment Information Transfer (TRAIT). Multivariable logistic regression modeling using the 11 resultant variables was employed to develop models in one cohort of first episode psychosis patients (n=297) and validate these models using data from a second cohort (n=309). The performance of the logistic modeling was good, with C-statistics ranging from .72 to .74 for the three outcomes. In the validation data the C-statistics were slightly lower, ranging from .67 to .71.

More recently we have examined the external validity of hospitalization as an outcome measure by showing that it is correlated with both a measure of global psychopathology, the Positive and Negative Syndrome Scale for Schizophrenia, which is widely used in clinical trials and the Quality of Life Scale. A manuscript reporting these findings has been accepted for publication (18).

Further Implementation

Development of these performance measures and data collection for pilot testing was funded by a Canadian provincial health services research fund. At present, our province, Alberta, and the other Canadian provinces lack the data infrastructure to implement these measures routinely. Currently, nationwide results on only 3 quality measures for mental health services are published by the Canadian Institute of Health Information (CIHI) which is an independent non-profit organization funded by both the Federal and Provincial Governments. The mental health indicators include only hospital-based indicators including:

• Rates of patient self injury
• Repeat hospitalizations for mental illness (the proportion of all patients hospitalized who have at least one other hospitalization within a year of the first hospitalization)
• 30-day readmission rate for persons hospitalized for a mental illness)

As an illustration of the advances in data collection that would be needed to implement these measures routinely, a "first psychotic episode" is not currently identified in either the national or provincial reporting systems. One way to achieve identification of the first episode would be to add that as a specifier to the Diagnostic and Statistical Manual for Mental Disorder classification system. At present this is not one of the modifications proposed for schizophrenia.

Despite these challenges, the development and validation of mental health performance measures remains an important enterprise because they are a necessary but not sufficient component of health services evaluation and quality improvement. Our local first episode psychosis health services research program is presently focused on two performance-measurement initiatives: first, testing the validity of hospitalization as a performance measure by linking it to other performance measures such as quality of life and relapse and second, developing an evidence based fidelity measure to assess adherence to evidence based practices in the treatment of first psychosis.

Author

Donald Addington, MBBS, MRCPsych, FRCPC
Department of Psychiatry, University of Calgary, Alberta, Canada

Disclaimer

The views and opinions expressed are those of the authors and do not necessarily state or reflect those of the National Quality Measures Clearinghouse™ (NQMC), the Agency for Healthcare Research and Quality (AHRQ), or its contractor, ECRI Institute.

Potential Conflicts of Interest

Dr. Addington reports business/professional affiliations with the University of Calgary and Alberta Health Services.

References

1. Adair CE, Simpson L, Birdsell JM, Omelchuk K, Casebeer AL, Gardiner HP et al. Performance measurement systems in health and mental health services: Models, practices and effectiveness: A state of the science review. Alberta, Canada, Alberta Heritage Foundation for Medical Research. 2003. 1-89.
2. National Committee for Quality Assurance. The state of health care quality 2006. Washington (DC), National Committee for Quality Assurance. 2006. 1-81.
3. Lehman AF, Steinwachs DM. Patterns of usual care for schizophrenia: Initial results from the schizophrenia patient outcomes research team (PORT) client survey. Schizophr Bull 1998;24:11-20.
4. Addington D, Addington J, Patten S. Relapse rates in an early psychosis treatment service. Acta Psychiatr Scand 2007;115:126-131.
5. Petersen L, Jeppesen P, Thorup A et al. A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. Br Med J 2005;331:602.
6. McCrone P, Knapp M. Economic evaluation of early intervention services. British Journal of Psychiatry Supplement 2007;51:s19-s22.
7. Bertolote J, McGorry P. Early intervention and recovery for young people with early psychosis: consensus statement. British Journal of Psychiatry Supplement 2005;48:s116-s119.
8. Addington D, Mckenzie E, Addington J, Patten S, Smith H, Adair C. Performance measures for early psychosis treatment services. Psychiatric Services 2005;56:1570-1582.
9. Addington D, Mckenzie E, Addington J, Patten S, Smith H, Adair C. Performance measures for evaluating services for people with a first episode psychosis. Early Intervention in Psychiatry 2007;1:157-167.
10. Addington D, Norman R, Adair C et al. A comparison of early psychosis treatment services using consensus and evidenced-based performance measures; moving towards setting standards. Early Intervention in Psychiatry 2009;3:274-281.
11. Craig TK, Garety P, Power P et al. The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis. Br Med J 2004;329:1-5.
12. Gleeson JF, Alvarez-Jimenez M, Cotton SM, Parker AG, Hetrick S. A systematic review of relapse measurement in randomized controlled trials of relapse prevention in first-episode psychosis. Schizophr Res 2010;119:79-88.
13. Burns T. Hospitalisation as an outcome measure in schizophrenia. British Journal of Psychiatry Supplement 2007;50:s37-s41.
14. Weiden PJ, Olfson M. Cost of relapse in schizophrenia. Schizophr Bull 1995;21:419-429.
15. Wunderink L, Nienhuis FJ, Sytema S, Slooff CJ, Knegtering R, Wiersma D. Guided discontinuation versus maintenance treatment in remitted first-episode psychosis: relapse rates and functional outcome. Journal of Clinical Psychiatry 2007;68:654-661.
16. Dixon LB, Adams C, Lucksted A. Update on family psychoeducation for schizophrenia. Schizophr Bull 2000;26:5-20.
17. Addington DE, Beck C, Wang J et al. Predictors of admission in first-episode psychosis: developing a risk adjustment model for service comparisons. Psychiatr Serv 2010;61:483-488.
18. Addington D, Mckenzie E, Wang JL. Validity of Admission to Hospital as an Outcome Measure for First Episode Psychosis Services. Psychiatric Services. [Accepted but not yet published]

Table 1: Performance Measures (PMs) for First Episode Psychosis

Table 2a: Performance Measures (PMs) (% conformance)

*Statistically significant p > 0.05

Table 2b: Performance Measure (PM) Results (Continuous Variable)