Moving beyond the shoestring: Highlights from a symposium to advance routine fidelity monitoring in Ontario's community mental health and addiction system

Fidelity assessment is a strategy to measure whether delivery of an intervention adheres to its original model or standards.

A recent symposium held in Toronto, Ontario, on November 12, 2019, brought key Ontario stakeholders in the community mental health and addictions (MHA) sector together to share experiences and discuss next steps for implementing routine fidelity monitoring in Ontario’s community mental health and addictions (MHA) sector.

They shared their experiences with fidelity monitoring and discussed next steps for implementing routine fidelity monitoring in Ontario’s community MHA sector.

This report includes highlights from this event and recommended next steps that emerged from the discussion.

Read the report below or download the PDF.

This report was prepared by Janet Durbin, Rossana Coriandoli, Avra Selick, Gord Langill

How to cite this document: Durbin, J., Coriandoli, R., Selick, A., & Langill, G. (2020) Moving beyond the shoestring: Highlights from a symposium to advance routine fidelity monitoring in Ontario's community mental health and addiction system. Centre for Addiction and Mental Health: Toronto.

Symposium planning committee

Dr. Janet Durbin, Provincial System Support Program (PSSP), Centre for Addiction and Mental Health (CAMH)

Gordon Langill, Canadian Mental Health Association, Haliburton Kawartha Pine Ridge branch

Avra Selick, University of Toronto (Doctoral trainee) and PSSP, CAMH

Debbie Bang, Addictions and Mental Health Ontario

Dr. Tim Aubry, University of Ottawa

Angela Yip, PSSP, CAMH

Kevin Barclay, Champlain Local Health Integration Network 

Dr. Yona Lunsky, Azrieli Centre for Adult Neurodevelopmental Disabilities and Mental Health, CAMH

Dr. Branka Agic, PSSP, CAMH

Dr. Karen MacCon, PSSP, CAMH

Dr. Chiachen Cheng, Northern Ontario School of Medicine

Main messages

Fidelity assessment is a strategy to measure whether delivery of an intervention adheres to its original model or standards. A recent symposium held in Toronto, Ontario, on November 12, 2019, brought key Ontario stakeholders in the community mental health and addictions (MHA) sector together to share experiences and discuss next steps for implementing routine fidelity monitoring in Ontario’s community mental health and addictions (MHA) sector. They shared their experiences with fidelity monitoring and discussed next steps for implementing routine fidelity monitoring in Ontario’s community MHA sector.

This report includes highlights from this event and recommended next steps that emerged from the discussion. 

What is the problem? 

  • Ontarians with MHA challenges are not receiving consistent high-quality care. 
  • Ontario is investing significant funds in the delivery of evidence-based models of care in the community MHA sector, but adherence to these models is not monitored. When these models are not delivered as intended, patients do not receive care that is based on the best available evidence. Without quality care, patients are less likely to achieve good outcomes. 
  • Currently, Ontario’s health care system monitors the volume of MHA services that are delivered but not the quality. 
  • Ontario has experience using fidelity assessments (e.g., Early Psychosis Intervention, Housing First, Strengths-Based Case Management), but projects have been poorly funded and time limited.

What is the solution? 

  • Fidelity assessments can address the quality gap by assessing whether delivery of an intervention adheres to its original model or standards. 
  • If routinely administered, they can inform efforts to shift current practice to higher quality across the health care system. Combined with outcome data, they can show if practices are achieving the intended outcomes for patients. 
  • Fidelity assessments have the most value when embedded in a comprehensive improvement strategy that also includes monitoring outcomes and providing implementation support to guide improvement work. This strategy needs to be supported with sustained funding.
  • Appropriate resources are needed so that existing and promising fidelity work in the province’s many MHA services can be sustained and spread. 

How can we get there? 

  • Ontario needs a dedicated, funded team or centre mandated to collect and report fidelity data and support quality improvement efforts. Fidelity assessments would be part of a comprehensive improvement model that also includes collecting outcome data and providing implementation support to guide improvement work. 
  • The centre would:
    • be responsible for conducting arms-length, unbiased fidelity monitoring, building agency capacity for fidelity monitoring, collating and sharing fidelity data, and adapting and developing fidelity scales and processes for Ontario
    • partner with and leverage the capacities and expertise of existing Ontario organizations (for example, those that provide standardized data collection, implementation support, and quality improvement training and reporting) 
    • have a multi-stakeholder governance structure to guide this work, including leaders from each sector and service users 
    • ensure feasibility, relevance, and rigour in the tools and strategies used for this work 
    • initially focus on one or two practices to build and refine their fidelity structures and processes, potentially based on the core basket of services
    • draw on current provincial fidelity experience and expertise and expand fidelity monitoring to other interventions as expertise and capacity develops across the province
    • include people with lived experience in this work. 
  • Similar centres have been developed in other jurisdictions and could inform this work (e.g., Centre for Excellence in Mental Health in Quebec, Centre for Practice Innovations at Columbia Psychiatry in New York State). 

Introduction

Transferring evidence-based care into routine practice is a significant challenge across jurisdictions. As a result, many patients are not receiving care based on the best available evidence. This is a problem in all areas of health care, including mental health and addictions (MHA). Systematic measurement of quality is required to address this issue. 

Fidelity assessment is a strategy to measure whether delivery of an intervention adheres to its original model or standards. Routine fidelity assessments can inform efforts to shift current practice to a more consistent, higher quality. When combined with outcome data, these assessments can show if practices are achieving the intended outcomes for patients. Overall, collection of these data can support a learning health care system. Service users can be more confident they are receiving good care that is in line with best available evidence, regardless of where the care is received, and Ontarians can have more confidence that health care dollars are being spent wisely. 

Ontario’s MHA strategy, Roadmap to Wellness, emphasizes the importance of using data to ensure consistent, high-quality delivery of evidence-based practices. Past Ontario reports have noted the lack of systematic quality monitoring in the community MHA sector. Fidelity monitoring offers a strategy to advance efforts to address this gap. In addition, experience and expertise in conducting fidelity reviews in community MHA programs has grown over the last few years. However, the projects have been poorly funded and time limited (for examples, see this backgrounder), and promising work has not been sustained or spread. If the full benefit of fidelity reviews is to be realized, fidelity monitoring needs to be implemented in a sustained manner with appropriate resourcing and needs to be part of a data quality strategy.

A recent symposium, funded by the Canadian Institutes for Health Research and held in Toronto, Canada, on November 12, 2020, brought close to 40 key Ontario stakeholders in the community MHA sector together to share experiences and discuss next steps for implementing routine fidelity monitoring in Ontario’s community MHA sector (see Appendix 1 for participating organizations). 

The symposium  presenters included:

  • Janet Durbin, Independent Scientist in the Provincial System Support Program, Centre for Addiction and Mental Health (CAMH)
  • Aristotle Voineskos, Director, Slaight Family Centre for Youth in Transition and Head, Kimel Family Translational Imaging-Genetics Laboratory, Campbell Family Mental Health Research Institute, CAMH
  • Kevin Barklay, Director, West Champlain sub-region at Champlain Local Health Integration Network (LHIN) 
  • Lisa Dixon, Director of the New York State Centre for Practice Innovation (CPI)
  • Shannel Butt, Coordinator, Prevention and Early Intervention Program, Elgin, and Co-Chair, Early Psychosis Intervention Ontario Network
  • Dr. Donna Pettey, Director of Integration, Research and Evaluation, Canadian Mental Health Association – Ottawa
  • Catharine Vandelinde, Executive Director, Options Bytown Non-Profit and member, Mental Health Commission of Canada team to implement Housing First across Canada

The symposium also included structured, facilitated table discussions to reflect on the presentations and consider relevant next steps for Ontario. 

This report provides selected highlights of what was learned during the symposium, focusing on experiences participating in fidelity assessments, and the recommended next steps for Ontario. The report begins with a brief overview of fidelity monitoring. 

About fidelity monitoring

What is fidelity measurement and why is it important?

The term “fidelity” refers to the extent to which delivery of an intervention adheres to established standards, guidelines or protocols. Fidelity measurement assesses the extent to which a practice has been implemented as intended. It is important because when an intervention that has been proven effective in research is transferred into routine practice, the expected outcomes will not be achieved if the intervention is not implemented as designed or not replicated with fidelity. Fidelity measurement is most powerful when it is embedded in a broader program of system improvement that includes routine assessment, feedback and implementation support. 

Routine fidelity monitoring in a system of care offers a number of benefits: 

  • The reviews can identify drift, or degrading, of delivery of an intervention over time. 
  • Fidelity scales articulate the core components of an intervention. As a result, they can provide a roadmap to guide high-quality implementation of new programs and expectations of existing programs. 
  • In a large system of care, fidelity monitoring increases the likelihood that clients will receive the same high-quality care, no matter where they access services.

Conducting fidelity reviews

There are many different approaches to conducting fidelity assessments (see Appendix 2). These vary in rigour, resource requirements, and distribution of burden (site and assessor roles). The gold standard approach is based on an in-person visit to a program by a team of trained independent assessors. The assessors observe program operations; interview staff, clients and families; review program documents (e.g., policies and manuals) and administrative data; and review client health records. At the end of the visit, they may also debrief with staff. After each site visit, the assessors prepare a detailed report with rating results, feedback about current practices, and quality improvement recommendations. 

The site visit method has many advantages, including the opportunity to build staff buy-in and the credibility of the review. Use of trained independent assessors enhances consistency and objectivity. However, site visits are resource intensive and alternative assessment approaches are being developed. Among these are remote approaches, where assessors gather site information through telephone interviews and reviews of program documents and data, minimizing travel and disruption to the site. Program self-assessment methods are also used, which have a lower cost but lack external checks. Assessments may be conducted using a hybrid approach: for example, a self-assessment followed by a brief visit from an assessor to review results.

The approach selection depends on a number of factors, including the available resources, the capacity of programs to collect and share data, and the purpose of the assessment. For example, fidelity assessments used for accountability require a different level of rigour than those used for program quality improvement.

Symposium highlights

From the field: Panel on fidelity experience and opportunities

Three senior individuals2 working in Ontario MHA sector organizations described their experiences receiving and conducting fidelity reviews. They addressed benefits and challenges and offered recommendations for future provincial fidelity work. This section provides an overview of their discussion.

Benefits associated with fidelity assessments:

  • The fidelity assessment provides an objective review of program practice in relation to the model’s expectations. It highlights gaps in service delivery as well as program strengths. 
  • The fidelity report can inform stakeholders about current practice, strengths and challenges, relevant for both program and system improvement. Stakeholders include program staff, senior leadership, funders, patients, and community partners. 
  • Feedback provides a “road map” for modifying and improving services and provides an opportunity for the fidelity team lead to problem solve with program staff.   
  • Fidelity assessments often include client input, providing a structured means for client feedback to inform improvement plans. 
  • Preparing for an assessment provides an opportunity for teams to review their program policies, procedures, and documentation practices. 
  • Fidelity assessments can reduce program drift and support consistent practice within agencies, across teams, and across the provincial system.
  • In instances where a peer fidelity model is used (i.e., clinicians from different programs are trained as assessors and evaluate each other), the assessors can learn from other teams and share clinically-based knowledge across programs. This is especially valuable for staff from small programs, who might be isolated and could benefit from opportunities to engage with other providers who deliver the same model of care. 

Risks or challenges associated with fidelity monitoring:

  • Program staff may perceive the review as an individual performance evaluation and feel uncomfortable or fearful. Engagement with the program and clear communication as to the purpose of the assessment are critical to avoid this misconception. 
  • Conducting fidelity assessments and implementing changes based on the fidelity report can be resource intensive for the program and organization. 
  • Fidelity assessments are generally voluntary and are not currently linked to funding or multi-sector service accountability agreements. 
  • Evaluation fatigue among frontline staff can make receptivity and participation in fidelity work difficult to sustain.  
  • For multi-program agencies, it can be difficult to coordinate multiple assessments that are not aligned. 

Recommendations for implementing fidelity monitoring in Ontario:

  • Implement a combination of annual self-assessments and external fidelity reviews every two years to help prevent practice drift. Self-assessments can be done in the context of an ongoing learning community.
  • Tie program funding to fidelity to ensure that high-quality practices can be sustained.
  • Build fidelity assessment processes that are flexible and allow newly emerging evidence to be incorporated. 
  • Build in feedback loops to ensure frontline staff have access to their own data and can see how fidelity feedback can inform quality improvement work and improved adherence to standards. 
  • Fidelity data should be rolled up and routinely reported across programs to allow for comparisons across programs and provide the opportunity for shared learning. 
  • Include people with lived experience who are well versed in the model as part of self-assessment teams to ensure programs meet the needs of those who receive services.

Government-supported intermediary organization: Centre for Practice Innovations 

Lisa Dixon3, the Director of the Centre for Practice Innovations, shared the Centre’s experience as an intermediary organization, conducting fidelity assessments in New York State. 

The Centre for Practice Innovations (CPI) at Columbia Psychiatry is funded by the New York State Office of Mental Health and is mandated to spread evidence-based practices for mental health and co-occurring addictions care within the state. CPI defines itself as an intermediary organization that is building the capacity of state providers and systems to implement and sustain best practice models.

CPI started in 2008 by supporting implementation and spread of two evidence-based practices (EBPs). Over time, it expanded and presently supports implementation of about eight EBPs, including assertive community treatment (ACT), Early Psychosis Intervention (EPI), and Supported Employment (see Appendix 3). CPI assists programs with preparation to implement the EBP, initial implementation, and maintenance through ongoing training, feedback, and technical assistance. 

CPI staff provide fidelity or performance measurement feedback support, depending on the program’s needs. CPI staff collect fidelity data through multiple methods, including self-assessments, administrative data, and site visits by independent assessors. The fidelity reports include performance ratings and improvement suggestions. Outcome data relevant to each practice are collected and reported to monitor improvement and flag challenge areas. Aggregated quarterly data allow comparisons among programs across the state. The reports are a foundation for discussions between program leadership and staff, and with CPI training teams, with a focus on how to improve low ratings. 

CPI central teams support implementation of each EBP and include fidelity coordinators, trainers, data management/analysis staff, and project managers. CPI’s work is informed by multiple stakeholder advisory committees, which include individuals with lived experience, families, and providers. 

Important to the work of the CPI is its emphasis on use of rigorous, yet feasible and relevant, approaches and methods, and ongoing improvement of these methods. The CPI has done extensive work to refine its methods for fidelity assessment and training, including by adaptating fidelity scales to the state system context and developing learning collaboratives (initiatives that bring peers together to study and apply quality improvement methods to a focused topic area) and distance-training technologies (see examples of CPI approach to fidelity assessment in Appendix 3).  

The CPI also conducts research on various topics including methods of fidelity review and implementation support, and practice improvement, supported in some cases through external grants. To support this work, experienced researchers are among the senior staff of the CPI. In addition to supporting program improvement, the CPI flags issues where state-level policy and resource changes are needed. 

Summary of table discussions

This section provides a summary of the responses of symposium participants to the questions posed during table discussions.

Should fidelity assessments be part of the Ontario data strategy?

Participants supported the inclusion of routine fidelity assessments in the Ontario data strategy and raised some important considerations and suggestions to advance implementation: 

  • Initiate fidelity assessments for EBPs with established fidelity scales while working to develop fidelity measures for EBPs that do not yet have any.
  • Implement routine fidelity measurement to help ensure that clients across the province receive the same services and the same quality of services regardless of where they live.
  • When selecting a fidelity assessment approach, consider the purpose of the assessment and related trade-offs, such as relying on administrative data versus site visits.
  • Make fidelity measurement a component of building quality care, combining it with, among other things, training, implementation support, and ongoing data collection and feedback.
  • Fidelity monitoring needs to be considered within the broader provincial data strategy, ensuring that it is feasible for programs among other monitoring and reporting demands.

What is working or not working about current approaches to fidelity measurement?

  • There is currently no system in place for routine fidelity assessments across community MHA programs. The result is that we know very little about the quality of care in the province. 
  • Our system has been using client volumes to determine agency funding, but this does not capture whether EBPs are being implemented with fidelity or provide measures that agencies can use to identify areas for quality improvement and sustainability. 
  • There has been a small number of time-limited fidelity initiatives in some sectors (e.g., ACT, EPI, Housing First) but these initiatives are voluntary and overburdened teams are not always able to continue participating in fidelity assessments over a long time. In the absence of routine assessments, practices may drift towards what the provider is most comfortable doing. 

What would an ideal fidelity assessment system look like in Ontario?

  • The Ontario government would implement a formal, centralized approach to fidelity monitoring, including a centralized, dedicated team or centre. 
  • The team or centre would be responsible for the following: conducting arms-length, unbiased fidelity monitoring; building agency capacity for fidelity monitoring, including providing technical support; collating and sharing fidelity data; adapting and developing fidelity scales for Ontario; and ensuring rigour and feasibility of methods.
  • Fidelity monitoring would be integrated with outcome monitoring and quality improvement supports to ensure data is used to improve quality of care for clients. 
  • People with lived experience and families would be engaged in the fidelity monitoring system from its inception.
  • Fidelity data would be reported to all stakeholders, including providers, clients, and families. 
  • Existing system resources, such as the Provincial System Support Program at CAMH, would be leveraged to support this work.
  • The resources required for agencies to participate in fidelity assessments (e.g., standardized documentation) would be built into agency budgets.  
  • Program funding would be tied to fidelity assessment results, with routine fidelity monitoring required as part of service agreements and funding letters. This would ensure that fidelity assessments would remain a core component of quality monitoring.
  • The new Ontario Centre of Excellence for Mental Health and Addiction would provide oversight and direction to the team or centre. 
  • A multi-stakeholder governance structure, including leaders from each sector and service users, would guide this work.

What are the first steps towards this goal? 

  • Conduct a targeted environmental scan and literature review of existing dedicated fidelity centres and how they work, methods for conducting fidelity, and existing standards and fidelity scales relevant to Ontario.
  • Develop recommendations for the Ministry of Health and Long-Term Care based on the environmental scan and literature review, including a model that incorporates fidelity into the existing quality improvement framework and includes capacity building, training, implementation support, and data collection. 
  • Start with programs or interventions that have an existing fidelity scale, such as Strengths-Based Case Management, EPI, ACT, and Housing First. Once more expertize and capacity is developed, scale up to other programs. 

Appendix 1: Participating organizations

  • Centre for Addiction and Mental Health
  • Options Bytown Non-Profit Housing Corp
  • Ontario Ministry of Health
  • Northern Ontario School of Medicine
  • Addictions and Mental Health Ontario
  • Ontario Association for ACT & FACT
  • Canadian Mental Health Association, Ontario Division
  • CMHA Toronto 
  • CMHA Elgin
  • CMHA Ottawa
  • CMHA Haliburton Kawartha Pine Ridge
  • Chatham-Kent Health Alliance
  • Mississauga Halton Local Health Integration Network
  • Champlain Local Health Integration Network
  • North East Local Health Integration Network
  • Community Care Information Management
  • Health Quality Ontario
  • Heads Up! Early Psychosis Intervention Program
  • Canadian Observatory on Homelessness 
  • Services and Housing in the Province 
  • Wilfrid Laurier University
  • Ontario Centre of Excellence for Child and Youth Mental Health
  • The Royal Ottawa 
  • University of Ottawa 

Appendix 2: Examples of fidelity assessment approaches

*Adapted from Essock & Addington, 2018

Assessment approach: In person site visit (external assessors)

Strengths

  • Considered the most rigorous. 
  • Allows collection of any type of data. 
  • Reduces reliance on program resources to collect or summarize data.
  • Enables assessment of qualitative aspects of the program (e.g., whether facilities are youth friendly).
  • Allows for direct contact between program staff and assessors, which may increase staff buy-in, credibility of feedback.
  • Provides a forum for interaction and clinical strategizing.

Limitations

  • Requires considerable resources (e.g., travel costs, assessor time).
  • Requires a pool of assessors trained to work. Independently in different locations and to score reliably.

Assessment approach: Remote assessment (Includes structured phone interviews, administrative data review)

Strengths

  • Can be deployed over large distances.
  • Involves central team of trained interviewers to enhance reliability.
  • Lowers the cost (no travel required).
  • Is less disruptive to the program (not necessary to host assessors on site).
  • Allows lower cost post-review feedback strategies to be incorporated (e.g., brief site visit; team feedback via web).

Limitations

  • Limits assessors’ access to data (e.g., direct observation of site context, such as youth friendliness and atmosphere of team meetings). 
  • Limits programs’ access to assessors.

Assessment approach: Administrative data

Strengths

  • Based on data for all clients (i.e., no need to sample). 
  • Requires minimal additional work, if embedded in routine program data collection.

Limitations

  • Requires programs to have capacity to collect, extract, and summarize data. 
  • Does not capture all relevant aspects of a practice.
  • Low data quality may be a challenge.

Assessment approach: Program self-assessment

Strengths

  • Is relatively low cost compared to other approaches.
  • Encourages team discussions and reflective practice.

Limitations

  • Relies on internal capacity and motivation of program to conduct assessment.
  • Is subjective and lacks external checks.
  • Allows limited opportunity for external input and discussion.

Appendix 3: Centre for Practice Innovations (CPI) Fidelity and other implementation support methods for three evidence-based practices

Assertive Community Treatment. CPI’s ACT Institute provides training and implementation support to the state’s 108 ACT teams. A blended-learning approach is used, as well as role training, learning collaboratives, support calls for unique roles, consultations, and technical assistance. 

CPI evaluators and ACT teams initially used the 47-item Tool for Measurement of Assertive Community Treatment. While staff found the fidelity self-assessment process useful, they found it to be time-consuming and not entirely compatible with state requirements. As a result, the CPI team worked with ACT clinicians and the developers of the fidelity tool to pare down the fidelity instrument to 27 items.

Individual Placement and Support (IPS). IPS is a supported employment program with a well-articulated model and a well-tested fidelity tool. The program includes use of fidelity and performance indicator data to drive continuous quality improvement efforts. CPI staff work with all rehabilitation programs and clinics across New York State to ensure that organizations implement IPS with fidelity. CPI hosts learning collaboratives and provides implementation supports and technical assistance.

CPI evaluators use the IPS Supported Employment Fidelity Scale and have demonstrated a high degree of concordance between programs’ self-assessments and expert assessments after one year of participation in the IPS learning collaboratives.

OnTrackNY (Early Psychosis Intervention). This coordinated specialty care program for people experiencing early psychosis has a centralized hub that provides training, funding, and evaluation. Data are gathered every three months on client demographics, hospitalizations, and engagement in treatment, work, or school.

The program fidelity scale was adapted from two existing scales, and the assessment was based on program data and a site visit. Each program team submits a quarterly fidelity self-assessment. A CPI evaluator then conducts a site visit with each program that includes interviews with staff, clients and family members; team meeting observation; and reviews of client charts and program records. The program leaders receive the final fidelity report, which includes details about the program’s strengths, key findings, and a score for each domain. A pilot study tested incremental value of the site visit and the refined tool to maximize efficiency while maintaining usefulness.

Notes

  1. Dr. Janet Durbin, Provincial System Support Program (PSSP), Centre for Addiction and Mental Health (CAMH); Rossana Coriandoli, Knowledge Broker, PSSP, CAMH; Avra Selick, University of Toronto (Doctoral trainee) and PSSP, CAMH, Gordon Langill, Canadian Mental Health Association, Haliburton Kawartha Pine Ridge branch.
  2. Panelists included: Shannel Butt, Coordinator, Prevention and Early Intervention Program, Elgin, and Co-Chair, Early Psychosis Intervention Ontario Network; Dr. Donna Pettey, Director of Integration, Research and Evaluation, Canadian Mental Health Association – Ottawa; and Catharine Vandelinde, Executive Director, Options Bytown Non-Profit and member, Mental Health Commission of Canada team to implement Housing First across Canada.
  3. Lisa Dixon, MD, MPH, Director, Center for Practice Innovations Division of Behavioral Health Services and Policy Research, New York State Psychiatric Institute Professor, Columbia University Medical Center.