Implementing a volunteer peer fidelity assessment in Ontario Early Psychosis Intervention programs: What did we learn?

In brief

The Early Psychosis Intervention Ontarioi Network (EPION)’s Standards Implementation Committee has released its fidelity pilot report. “Implementing a volunteer peer fidelity assessment in Ontario Early Psychosis Intervention programs: what did we learn?” describes the fidelity pilot project conducted by the Standards Implementation Steering Committee (SISC).

The fidelity pilot project was initiated to test the feasibility and value of using an innovative peer assessment method to assess the fidelity of selected Ontario EPI programs. Peer assessor teams of volunteer EPI program staff and CAMH evaluators were formed to conduct the program reviews. The pilot was completed in nine programs.

Project objectives:

  • Develop and implement a peer fidelity assessment process with a subset of Ontario EPI programs.
  • Evaluate the assessment process, considering consistency, feasibility, and value to programs.
  • Review fidelity results, considering how they could be used to support program and sector improvement.
  • Make recommendations for future fidelity monitoring.

This fidelity pilot report describes the review process and the results of the evaluation. Also included are recommended next steps related to fidelity monitoring in EPI programs.

In this Research Report Round-up, we present a summary of the report. Research Report Round-ups are brief summaries of research reports, presented in a user-friendly format.

Read the summary below or download the PDF.

Title and link to report: Implementing a volunteer peer fidelity assessment in Ontario Early Psychosis Intervention programs: What did we learn?

Authors: Janet Durbin, Avra Selick, Gordon Langill, Chi Cheng, Suzanne Archie, Donald Addington, and the Early Psychosis Intervention Ontario Network (EPION) Standards Implementation Steering Committee.

Date of release: September 2018

Location: Toronto
Language: English

What this report is about

facilitator working with adult traineesRoutine fidelity monitoring can help programs ensure they are following pre-set standards. Ontario community mental health programs do not routinely do this type of assessment. This project tested the feasibility and value of a volunteer peer fidelity assessment process for Ontario Early Psychosis Intervention (EPI) programs.

Key components of the process included:

  • Three-person teams (two volunteer peer assessors from EPI programs and an evaluator from CAMH);
  • Use of the validated First Episode Psychosis Services Fidelity Scale;
  • Visits to the program sites to learn about EPI delivery; and,
  • Data collection from participants about feasibility and value of the process.

A central team provided training and coaching to the assessors with the aim of increasing the consistency and accuracy of the fidelity ratings. Nine programs from across Ontario were assessed and 20 individuals were trained to conduct the assessments.

Many elements of the assessment process supported the assessors to produce consistent and accurate ratings, as well as narrative assessment feedback to programs, and to gather  information about the process itself:

  • The assessor team composition worked well; the EPI peer assessors had credibility with the programs and understood program practices; CAMH staff brought knowledge of program evaluation and of systematic data collection.
  • The site visits were useful in helping assessors learn about each program.
  • The detailed manual, training and coaching provided by the central team guided all components of the process.
  • Charts, surveys and focus groups with participants provided feedback about the feasibility and value of this fidelity assessment process.

How EPI program participants felt about the process:

  • The fidelity reviews were valuable.
  • They appreciated the feedback on program strengths and challenges, and the improvement suggestions (which some have started to act on) were useful.
  • The common assessment has created an opportunity for programs to share practices and collaborate on improvement projects.

What was learned about fidelity in program delivery:

  • Programs received satisfactory or exemplary ratings for their performance on many model elements, although there was variation. One area where programs tended to receive lower ratings was in the psychosocial treatments domain (e.g., cognitive behaviour therapy, weight management, substance use treatment). Assessors suggested that programs provide more consistent documentation of the care they deliver and use manuals and protocols to standardize how and when they provide services.

What was learned about feasibility:

  • A number of feasibility issues emerged. It took time for assessors to learn how to do assessments. Even in this time limited pilot  there was assessor turnover and, going forward, it will be a challenge to maintain a pool of qualified assessors. In addition, central supports are required to organize the assessments, deliver assessor training, and aggregate fidelity results.
  • There was value to using an existing validated tool, but it needs some refinements to be relevant to the Ontario service context. The project team has started work to develop supplementary items specific to the Ontario EPI Program Standards.

Next Steps:

  • Based on the positive feedback, the project is conducting a second round of assessments, with some minor changes to increase efficiencies while staying true to the peer team model.
  • Sustaining and spreading this work would benefit from a commitment of funding and support from health system stakeholders.

Recommendations for the EPI sector:

  • Conduct a follow-up survey to assess the impact of the fidelity assessment on the quality of services on the programs that participated in this project.
  • Continue to work towards routine sector-wide fidelity assessments in Ontario EPI programs.
  • Explore strategies to increase efficiencies and reduce the burden on assessors and programs (e.g., alternate in-person assessments with self-assessments; incorporate remote assessment strategies).
  • Explore opportunities to support the use of fidelity results to improve quality of EPI services, such as through quality improvement projects and engagement with system partners involved in quality improvement work.
  • Explore other funding strategies to help make routine fidelity assessments in EPI programs more sustainable (e.g., fee-for-service funding, partnerships with other funding organizations).
  • Engage other sectors to learn from their experiences with implementing fidelity assessments.

How can this report be used

This report offers recommendations and strategies to increase the use, and enhance the quality, of fidelity assessments for EPI programs. It should be of use to system and program planners, and providers of EPI services, as well as policymakers, to implement routine sector-wide fidelity assessments in Ontario EPI programs.

Type of study

Researchers developed and implemented a peer fidelity assessment process with a group of Ontario EPI programs. Qualitative feedback was obtained from assessors and programs to learn about the consistency, feasibility, and value of the process. Fidelity results were reviewed with consideration of how they could be used to support program and sector improvement, and future fidelity monitoring.

Key words

Early psychosis intervention; fidelity; assessment; monitoring; program standards; quality improvement.

Contact person/source

Janet Durbin, Independent Scientist, Data Evaluation Leadership Transformative Analyses (DELTA), Provincial System and Support Program (PSSP), CAMH at janet [dot] durbin [at] camh [dot] ca
Avra Selick, Research Specialist, DELTA, PSSP, CAMH at avra [dot] selick [at] camh [dot] ca.