How well are we doing? Measuring fidelity in Ontario’s early psychosis intervention programs

What you need to know:

Fidelity assessments are a strategy to monitor quality of care in relation to expected practice.  Using the 31 item First Episode Psychosis Service Fidelity Scale (FEPS-FS), researchers assessed nine early psychosis intervention (EPI) programs in Ontario to find out how well they follow the international EPI standards. Overall, the programs were rated as satisfactory or exceptional in many areas. Some common challenges were identified where system supports may be needed as well as issues that can be addressed at the program level. Results suggest that routine fidelity assessment can be a valuable data source for system improvement if feasible strategies for conducting the reviews can be developed.

 

This Research Snapshot looks at the article, “Using Fidelity Measurement to Assess Quality of Early Psychosis Intervention Services in Ontario,” published in the journal Psychiatric Services in 2019. Read it below or download the PDF

What is this research about?

Achieving high-quality implementation is important to provide effective and consistent delivery of evidence-based health interventions. However, it can be difficult to measure fidelity to an evidence-based model due to insufficient resources for routine monitoring  and a lack of tools that define the core components of a model of care along with performance criteria.

Early psychosis intervention (EPI) programs follow an evidence-based model of care that includes a team providing pharmacological and psychosocial interventions to support clients through recovery from a first episode of psychosis. In 2011, the Ontario Ministry of Health and Long-Term Care developed EPI Program standards. An EPI program fidelity measure that aligned with the Ontario Standards was published in 2016.

close up of someone filling out a paper formWhat did the researchers do?

Researchers worked with a volunteer sample of nine EPI programs across Ontario, recruited through an email call out to the Early Psychosis Intervention Ontario Network. They used the First Episode Psychosis Service Fidelity Scale (FEPS-FS) and rated EPI implementation across 31 items using a 5-point scale. A rating of 4 on an item indicated satisfactory performance.

Assessments included a program site visit by trained volunteer, peer assessors.  The assessor teams included two EPI program staff and one evaluation specialist. During the two-day site visits, the assessors:

  • interviewed staff, clients, and families
  • reviewed client charts
  • observed a team meeting
  • reviewed all program documents, and
  • examined administrative data.

 

Assessors developed preliminary ratings for each of the scale items, held consensus meetings with a fidelity expert to finalize the ratings and reported their findings back to the programs, with recommendations for improvement.

What did the researchers find?

Overall, the nine programs scored an average of 3.1 to 4.4 across the rated items and five programs exceeded a satisfactory performance rating of 4. Highest rated were items that assessed structural components of the model (e.g., “patient–provider ratio” and “assigned a case manager”) and prescribing practices. The lowest ratings were for items assessing provision of psychosocial treatments, including cognitive–behavioural treatment, supported employment, weight gain prevention, and substance use treatment. 

Common challenges across programs were lack of manuals and protocols to standardize delivery, and inconsistent documentation. Smaller programs scored lower on weekly meetings and multidisciplinary teams, along with some psychosocial care items.

Limitations and next steps

The results of this study may not reflect the broader EPI sector in Ontario because of the small sample size and self-selected participation. While efforts were made to enhance rating consistency, inter-rater reliability was not assessed. Since Ontario standards include items that are not assessed in this scale (for example peer support), and the international scale includes items that are not included in Ontario EPI standards, further work is needed on the validity of the scale for Ontario practice.

How can you use this research?

This research can be used to help EPI programs routinely assess how well they are following EPI standards. Other sectors can also learn how to conduct fidelity assessments using volunteer peer assessors.

About the researchers

Janet Durbin,1 Avra Selick,1 Gordon Langill,2 Chiachen Cheng,3 Suzanne Archie,4 Shannel Butt,5 Donald E. Addington6

1. Provincial System Support Program, Centre for Addiction and Mental Health, Toronto, Ontario
2. Canadian Mental Health Association (CMHA), Haliburton, Kawartha, Pine Ridge, Ontario
3. Psychiatry Section, Clinical Sciences, Northern Ontario School of Medicine, Thunder Bay, Ontario
4. Cleghorn Early Intervention in Psychosis Program, St. Joseph’s Healthcare Hamilton, Hamilton, Ontario
5. Prevention and Early Intervention for Psychosis Program (PEPP), CMHA Elgin Branch, St Thomas, Ontario
6. Department of Psychiatry, Foothills Hospital, University of Calgary, Calgary, Alberta

Keywords 

Fidelity, measurement, assessment, quality, improvement, early psychosis intervention, Ontario, scale, monitoring, implementation

This Research Snapshot is based on the article, “Using Fidelity Measurement to Assess Quality of Early Psychosis Intervention Services in Ontario,” published in Psychiatric Services in 2019. https://doi.org/10.1176/appi.ps.201800581.