Evaluation Tools and Guides

The Evaluation Framework provides Governments and other key stakeholders with tools and a series of guides to assess the effectiveness and cultural responsiveness of proposals and to further progress the implementation and evaluation of programs, services and resources.

The following series of guides outline:

  • The relevant methodologies to conduct culturally appropriate and comprehensive assessments of programs, services and interventions with regard to rigour, program impact, and community and stakeholder responses to the suicide prevention programs
  • Culturally relevant indicators to measure the effectiveness of programs and services addressing social and economic determinants on Indigenous social and emotional wellbeing outcomes
  • The steps and key considerations for communities and organisations to plan, implement and evaluate their own community level suicide prevention activities; for governments and program funders to evaluate suicide prevention proposals; and for health practitioners to evaluate clinical services, protocols and practices.
  • Tools and resources relevant to each of the guides.

Key Concepts in Conducting Evaluations

The most effective suicide prevention involves both whole of community and whole of government approaches. A whole of government approach encourages and facilitates the implementation of comprehensive suicide prevention programs through the development of program linkages and partnerships between government service providers, Indigenous organisations and peak bodies in the delivery of Indigenous programs and through the specification of the goals in program evaluation criteria.

The following guides include key considerations and to assist communities and governments to develop genuine partnerships and dual accountability measures to plan suicide prevention initiatives and to assess the effectiveness, cultural appropriateness and the efficiency of delivery of specific programs and program providers.

From a government and Indigenous leadership perspective, positive change requires system level change based on evidence of ‘what works’ to promote effective and sustainable suicide prevention. The findings of ATSISPEP literature review confirm that to effect system level change to reduce suicide and self-harm will require:

  • Genuine engagement with Aboriginal people, communities and organisations to promote sustainable, culturally responsive and effective services and programs
  • Greater interagency understanding of, and commitment and capacity to address, the variability of human capability and community readiness in Indigenous populations as a consequence of risk factors such as poverty, illness, incarceration, low self-esteem, trans-generational trauma and its relationship to suicide and self-harm
  • Greater support for families and communities by directly strengthening the protective factors and implementing programs that contribute to pathways to recovery
  • Resources and strategies that strengthen the commitment, understanding and capacity among mental health, health, education, justice, child protection and social services and their staff
  • Strategies and policies that are designed to reform and transform systemic, organisational and individual racism and discrimination and contribute to the development and embedding of cultural capabilities and competence across all relevant services and professions (Calma & Dudgeon, 2009; ATSISPEP, 2015; Walker, 2010, Walker et al., 2014)
  • Workforce development, Indigenous health curriculum and Indigenous governance and leadership
  • Primary and tertiary services that are culturally responsive, flexible increase access and integration and partnerships

The CBPATSISP Evaluation Framework provides an overarching mechanism to demonstrate the extent to which these elements are implemented, monitored and evaluated by the relevant government and non-government agencies to successfully achieve goals of the Fifth National Mental Health and Suicide Prevention Plan, relevant jurisdictional implementation plans and the renewed Social and Emotional Wellbeing Framework 2017-2023.
This requires State and Commonwealth Governments to genuinely engage with Indigenous Australians to develop local, culturally responsive approaches to identify and respond to individuals, families and groups most at risk within Indigenous communities, and support the evaluation of the cultural responsiveness and effectiveness of programs initially funded through the National Aboriginal and Torres Strait Islander Suicide Prevention Strategy, and now Primary Health Networks (PHNs).

Government and industry organisations generally specify a range of Key Performance Indicators to assess the effectiveness, (cost effectiveness), efficacy, efficiency and cultural responsiveness and safety of programs they fund. Assessing the effectiveness of implementing suicide prevention programs and strategies is a rigorous approach to ensure that what a community or organisation is doing or plans to do actually works.

In Aboriginal and Torres Strait Islander contexts it requires the following steps or considerations:

  • Identifying which strategies will be most likely to reduce self-harm and death from suicidal behaviour
  • Determining the potential effects of cultural, social, historic, legal, ethical, and economic factors on those strategies
  • Determining the most effective, culturally responsive and culturally safe methods for implementing those strategies
  • Assessing the effectiveness of the strategy in achieving its intended outcomes as it is developed and implemented

The CBPATSISP Evaluation Framework outlines steps to conduct formative, ongoing, continuous and summative stages of evaluation, process and outcome approaches throughout the development, implementation and evaluation of the project. It includes questions and mechanisms to assess the cultural responsiveness and effectiveness of initiatives at each of these phases. It includes a set of questions to be asked in determining the relevance of a proposed program, service or initiative and their alignment with the goals of relevant government strategies. It also includes a set of questions to determine the extent to which the project or activity can provide evidence that the longer term benefits of the outcomes are likely to outweigh the costs.

The higher the level of risk of the target population, the more intensive the prevention effort is required and the earlier it needs to begin. The following considerations are crucial for policymakers and program funders in determining both the effectiveness of suicide prevention strategies and policy and funding priorities when making decisions regarding:

  • Potential to reduce or avoid self-injury or death
  • Social, legal, and ethical and economic impact
  • Best methods to ensure cultural safety

Funding agencies, service providers and communities also need to know ‘what works’ in developing/adapting and implementing suicide prevention programs and services to address community identified needs and priorities outlined in the Solutions that Work (2015). This requires being able to identify the features of effective, culturally responsive programs, services and initiatives in order to achieve successful outcomes.

The most important things funding providers need to consider in the first instance include:

  • The delivery approach of the program or initiative
  • The formation of partnerships
  • Integration and co-ordination of interagency and inter-sectoral services
  • Capacity building (of staff and of groups /communities intended to benefit)
  • Targeting approaches (which population group/s is/are involved? and whether universal, indicated or targeted approaches to suicide prevention are required)
  • The use and quality of evidence and research in establishing the rationale for the proposed initiative

Questions

  • What works best for whom, why and when?
  • Does the service, program or initiative help achieve outcomes in other relevant and related program areas? (eg. Are there links with the drug and alcohol strategy and/or through interagency and/or other government programs?)
  • Does the service, program or initiative include strategies to address the local contextual issues that may hinder the effective implementation of key Aboriginal and Torres Strait Islander priorities in the Fifth Mental Health and Suicide Prevention Plan in achieving its objectives and outcomes?
  • Does the program or activity include evidenced based ‘what works’ elements that are likely to make a positive difference as outlined in the ATSISPEP Solutions?

That Work Report

  • Is there a sound theory or evidence base (which addresses social determinants and casual pathways) to help explain why some initiatives are most likely to work?
  • To what extent does the program or activity take account of the key principles underpinning the CBPATSISP Evaluation Framework and key elements identified in community consultations (see table 1)?
  • Is there an evidence-based rationale to demonstrate how the goals and objectives of the project/activity are designed to address identified risk and protective factors for a specified group, local community or whole population? And if so,
  • Is there a sound statistical data and evidence base to inform the selection of the project, program and activity?
  • Is there evidence of strong community consultation and leadership in determining the issues and priority area being addressed and in implementing the program?

The following considerations are crucial in assessing the extent to which the goals, aims, purpose and outcomes of suicide prevention and early intervention programs, services and initiatives for the specific group intended to benefit are cultural appropriate. The considerations are equally important for mainstream, universal programs (including Indigenous community) as they are for indicated and targeted programs and services.

Questions those undertaking the evaluation (including in the formative phase of development) need to ask to what extent:

  • Does this program/initiative/service focus on all or some aspects of the physical, spiritual, cultural, emotional and social wellbeing of an individual, their family and community?
  • Does this program/initiative/service focus on all or some aspects of the physical, spiritual, cultural, emotional and social wellbeing of an individual within specific risk groups (ie LGBTQI)?
  • Do workforce initiatives encourage and resource frontline workers and practitioners to focus on the physical, spiritual, cultural, emotional and social wellbeing of the individual, family and community?
  • Does the program/initiative/service or workforce initiative/resource recognise trans-generational trauma and align with the social and emotional wellbeing principles and principles of empowerment?
  • Does the program/initiative/service aim to strengthen cultural values and commitments, family and kinship systems of care, and Indigenous control and responsibility as an intrinsic aspect of healing and facilitating cultural, social and emotional wellbeing?
  • Does the service/program or workforce initiative acknowledge and work in partnership with the Aboriginal and/or Torres Strait Islander community-controlled sector and facilitate Indigenous people’s right to determine the types of services they receive?
  • Does the service or program or workforce initiative have a statement of commitment and strategies to address racial discrimination?
  • Does the program/initiative support human rights and social justice principles?
  • Does the program/initiative support Indigenous organisations cultural governance/self determination?

Program and service providers or external evaluators undertaking the evaluation (including in the formative phase of development) need to work with Aboriginal and Torres Strait Islander people (as co-designers) assess the quality of the program plan including the evaluation to build in key principles and culturally appropriate methods and methodologies from the onset.

A range of risk factors can impact on the sustainability of a program or initiative regardless of its effectiveness, including the short-term nature of most funding programs, difficulties in obtaining and retaining suitable personnel and poor participation rates due to lack of transport and family constraints, lack of community involvement in the program development. Many initiatives remain as pilots and are not adequately evaluated.
Community ownership of, and genuine partnership and contribution throughout all stages of the development and implementation process are fundamental to the program and can help offset these issues. In order to assess the extent to which community controlled or government or non-government organisations delivering a program or initiative actively took steps to achieve sustainability it is important to consider the following questions:

Questions

  • Does the project or initiative ensure a culturally responsive, localised and stable Indigenous control and governance model?
  • Is there a long-term vision, leadership and support for the specific initiative, project or activity within a community, region or state?
  • Has there been broad and ongoing Indigenous community consultation and strategic planning?
  • Is there evidence of genuine engagement with community groups and the community controlled health sector as well as with relevant stakeholders and institutions?
  • Is there evidence of collaborative activities being undertaken to support building, linking and leveraging relevant relationships and programs with partner organisations to strengthen sustainability?
  • Have funders contributed to constructive and preventive ‘upstream’ investments to achieve positive goals that contribute to Indigenous wellbeing, promote resilience and reduce suicide?
  • Have program funders and service providers a commitment to incorporating an appropriate set of indicators to evaluate the effectiveness of a program or initiatives’ short, medium and long term outcomes?

Community Guide to Planning and Evaluating Community-Based Initiatives

This section outlines steps for community groups, community controlled organisations and services to work in partnership to plan, implement and evaluate suicide prevention initiatives using Participatory Action Research (PAR) processes to ensure that programs are responsive to the local culture, context and specific issues.

PAR is a key component in planning, developing and delivering effective community-led suicide prevention programs and services for Indigenous peoples. It involves:
1) Developing a culturally safe environment for program participants
2) Fostering trusting relationships between the community and stakeholders

Establishing trust and building respectful relationships and genuine partnerships is an important first step in gaining community support, facilitating community ownership and engagement, and the broader community for, and uptake of, suicide prevention programs and services.

PAR provides a mechanism to integrate the cultural and social relevance for communities with rigour. The demonstrated benefits of using PAR include:

  • Strong engagement of the community by way of increased participant recruitment and program retention rates
  • Validated cultural measures
  • Reduced reporting bias
  • Enriched interpretation of research findings
  • Increased translation of findings into action
  • Greater ownership of the research and immediate transfer of knowledge into practice

Community development and PAR provide an important information resource and an enriched evidence base for engaging in community-wide planning, identifying relevant programs and initiatives and involving families and communities in developing, implementing and evaluating programs and services. These methods also enable national and state jurisdictions and funding bodies to support local, community-driven suicide prevention strategies and plans.

Key benefits of implementing the CBPATSISP Evaluation Framework using these culturally appropriate and responsive methods and engagement with the CBPATISP website include enabling communities to:

  • Build a local and national evidence base drawing primarily on Indigenous knowledge, experience and perspectives
  • Undertake comprehensive assessments of the overall comparative efficiency of initiatives
  • Produce information about ‘what works for whom and why’ to guide policy development, initiative and project implementation
  • Provide evidence of existing gaps to reform reporting and accountability requirements
  • Promote learning by communities /organisations proposing or implementing projects
  • Document and share with stakeholders emerging understandings of barriers, enablers and effective strategies and good practices in suicide prevention, early intervention and postvention
  • Provide compelling evidence of what does not work for Indigenous individuals, families and communities as well as what impacts adversely on Indigenous controlled services or training programs.

Evaluations that identify ‘what is working’ and how to enhance and continuously build on program and service outcomes can directly enable greater effectiveness of programs.

The main steps involved in establishing a culturally responsive evaluation process are outlined in Figure 1. These processes provide an opportunity for community groups and community controlled organisations (often in partnership with program and service providers) to use community development and PAR approaches in the planning, implementation, monitoring and evaluation of suicide prevention activities.

These processes enable communities (and partnership agencies and services) to regard their community suicide prevention initiative as a unique opportunity to bring about profound and positive change for Indigenous communities.

The teams planning and conducting evaluations are encouraged to:

  • Use a qualitative yarning approach involving Indigenous researchers/consultants for data collection, interpretation, analysis and reporting
  • Give priority to Indigenous perspectives, values, knowledge and experiences using in-depth unstructured conversational interviews and facilitated focus group discussions
  • Collaborate with existing project partnerships using consultation processes and agreements established with Indigenous communities throughout the project development and implementation

The actual specifics of the evaluation methodology and indicators, specific questions and data collection and communication reporting format need to be negotiated with Indigenous communities at an early stage, taking into account the local community contexts.

The key tasks for a culturally responsive community planning and evaluation process include:

  1. Establishing a local reference group with relevant community organisations and members who have relevant knowledge, skills and understanding to provide additional advice on the evaluation process
  2. Negotiating to include appropriate qualitative and quantitative methodologies and reporting arrangements with all relevant stakeholders groups including Indigenous partner organisations, community leaders and stakeholders
  3. Building a comprehensive understanding of the key issues and factors in the specific community contexts for specific populations of concern (eg. children, young people, men, families, perinatal mental health, people in incarceration, young people in out of care, and LGBTQI or other vulnerable groups)
  4. Building an understanding of the risk and protective factors around suicide generally and in local context
  5. Referring stakeholders to the CBPATSISP website resources and information on what is available to assist communities and services
  6. Reviewing and analysing relevant background reports and other relevant documentation regarding the project
  7. Reviewing literature relevant to building strong Indigenous families and communities
  8. Reviewing literature on ‘what works’ in suicide prevention and postvention
  9. Consulting with relevant governmental staff and other stakeholders at the community level
  10. Working collaboratively with groups to identify and conduct appropriate ‘information rich’ case studies using culturally responsive methods (ie storytelling, yarning, focus groups etc)
  11. Disseminating draft findings and recommendations to community and relevant agencies using appropriate information formats for different audiences (ie community feedback sheets, project newsletters, fact sheets and policy briefs for policy and funding personnel). Figure 2 provides an overview of the steps involved in developing a community-initiated/led, whole of community suicide prevention initiative.

DEVELOPING, IMPLEMENTING AND EVALUATING COMMUNITY SUICIDE PREVENTION

Using the CBPATSISP Evaluation Framework in developing, implementing and evaluating programs, services or specific initiatives (universal, targeted) teams need to consider the extent to which the community engagement and planning processes are:

  • Culturally responsive and secure to support Indigenous peoples’ involvement and leadership in all phases of the initiative
  • Incorporate the key guiding principles
  • Use local language in framing the evaluation.

The next section includes a set of suggested questions in each of the planning, implementation phases to support and assess the development, implementation and evaluation of intended outcomes.

Step 1. The first step for community researchers, groups, organisations (and services) involves:

  • Bringing community stakeholder together to share their stories;
  • Defining the issues surrounding suicide and self-harm or suicidal risk in their community;
  • Identifying existing services or programs and their roles in suicide prevention and/or crisis intervention or postvention.

Considerations/Questions

  • What are the issues being faced by the community?
  • Why do community members and stakeholders think these issues are happening?
  • What do people think needs to happen to address these issues?
  • What are the potential barriers and enablers for this to happen?
  • What does the new or existing program or initiative hope to achieve?
  • What services and agencies need to be involved?

Step 2.

Draw a diagram to show the major relationships between groups and issues in the story

Describe how activities relate to desired outcomes (eg. establish strong and secure early years programs, increase young people’s sense of identity, strengthen the role or fathers, mothers and families, provide hope for the future, and connect people to culture and country).

Step 3.

Review relevant examples of community resources for planning and evaluation in suicide prevention listed in the CBPATSISP resources section and on the ATSISPEP website.

Step 4.

Implement the evaluation plan.

Step 5.

Identify assumptions about relationships between activities and outcomes from an Indigenous community’ perspective, make sure all relevant groups and individuals are included in building the picture using culturally responsive and respectful processes and protocols.

Considerations/Questions

  • How do these ideas/perspectives match/align with existing government program goals and objectives and expected outcomes?

Develop the key elements the community or organisation wants to explore as the story unfolds in order to develop key questions and/or evaluation/value statements.

This phase involves drawing together sound/robust observations and interpretations of individual and communal experiences using a range of approaches to explore issues and begin to gather both qualitative and quantitative information using various methods. It requires identifying the sources and timeframes for collecting data and developing a plan and set of tables to guide the implementation of the evaluation.

Questions might include:

  • What are the key issues that matter to individuals, families and communities?
  • How can they be included in the program?
  • How can they be measured and costed?
  • Are local issues/ potential barriers taken into account?
  • Are all groups in the community involved?
  • Is/are local language/s recognised, are all family groups and language groups represented?
  • Are key people involved?

Consider issues of cultural and community appropriateness in collecting and interpreting information, using approaches that have been identified as appropriate and supporting Indigenous terms of reference including PAR, as well as assessment tools to measure empowerment and change such as Most Significant Change Technique (MSCT), appreciative inquiry, photovoice, art appreciation; bricolage, Growth and Empowerment Measure (GEM) (Haswell et al., 2013) and other community audit tools.

This step involves working with relevant local Aboriginal and Torres Strait Islander groups to assist in interpreting the formative information using appropriate techniques for analysing the qualitative and quantitative data (eg thematic narrative, NVivo).

PAR and Continuous Quality Improvement (CQI) can be applied using the knowledge and findings generated through the ongoing evaluation throughout the implementation to improve programs, services and practices. The same processes are used in establishing the summative evaluation findings to meet the different needs of multiple audiences – including the community, policy makers, program and service providers, other relevant stakeholders and the wider community. In this instance the findings and recommendations can be used to support the continuation of funding and implementation of the programs.

This phase seeks to obtain data related to the following major themes:

Questions

  • What resources have been allocated towards the sustainability of project processes and activities – now and in the future?
  • What strategies have been put in place to ensure sustainability, enhancement or further development of project outcomes?
  • What are the key features of the organisation that impact on success? Taking into account the extent to which these:
    a) affect the success of the project
    b) are affected by planning and implementing the project
  • How effective are the partnerships to the success of the project/services? – the nature and activities of partnerships involved in the projects, the role in facilitating the formation of community partnerships and partnerships with community controlled organisations in the health, mental health and/or social and emotional wellbeing services sector, and partnerships between agencies
  • What has changed as a result of the project? – positive, negative and unexpected outcomes, outcomes as they relate to the overall policy outcomes, participant satisfaction (individual, family and community level)
  • Why did it change as a result of the project? – participant perspectives regarding the role of the intervention program or service in achieving policy outcomes, using narrative storytelling, MSCT, photovoice and other culturally relevant data collection methods
  • What activities were undertaken by projects? – and their cultural relevance and importance in contributing to the outcomes
  • What other external factors contributed to the outcomes? – and importance of those other factors (e.g. other activities undertaken concurrently by the auspicing agency, previous activities, activities of other agencies)
  • What were the barriers and enablers to the success of the project? – including features of
    a) the community (e.g. community readiness, level of community capacity, previous experience with relevant projects)
    b) the local context – local conditions
    c) the project (e.g. its staffing, levels of cultural competency and previous experience, support from auspicing organisations, support for other agencies)
  • What are lessons learnt and next steps? – using narrative storytelling, MSCT, photovoice and other culturally relevant data collection methods to consider what participants and personnel think could have been done different and more effectively or efficiently to achieve intended outcomes
  • Other comments? – an opportunity for project participants to comment on anything else of importance that has not been collected in relation to the framework

An important aspect of any evaluation is to celebrate the story of ‘what works’ with the community, as well as reflecting on and learning from those aspects of the story that don’t work, and using the knowledge created through the evaluation to improve programs and services moving forward.

The evaluation can also be used to leverage more funding and to identify what further research is needed to fill gaps in knowledge about suicide prevention in Indigenous contexts.

Examples of promising programs, initiatives and services identified as demonstrating best practice initially identified through the ATSISPEP have been further reviewed and are available on the CBPATSISP Clearinghouse website.

In accordance with the Evaluation Framework principles, the first step in establishing strong healthy communities is to acknowledge and understand the devastating and enduring impacts of colonisation on Indigenous people’s contemporary lives. It also requires understanding the various pathways necessary for healing from historical trauma, using both cultural and contemporary understandings and processes to support social and emotional wellbeing. Establishing culturally responsive community, family and individual support systems and programs to promote pathways to recovery through dealing with loss, grief and disconnection, trauma and helplessness, and powerless and lack of control is essential to reduce mental health issues, high risk behaviours and suicide. This in turn requires:

  • Self-determination and community governance
  • Reconnection and community life
  • Restoration and community resilience

Although there is considerable overlap within the three themes, it is important to address each one specifically as well as all three collectively to promote healing and recovery across generations. The interrelatedness of the ways in which the ongoing history of transgenerational trauma impacts on individual, family and community social and emotional wellbeing, and specific pathways to recovery, are depicted in Figure 24.2 below (Milroy, Dudgeon & Walker, 2014, p. 424).

The diagram shows how each of the pathways to recovery outlines a conceptual framework and theory of change for program development and, in combination, provide a holistic framework to support community life and development, which include both the types of questions required to be addressed in developing, implementing and evaluating programs and services and the types of promising programs that are useful for community groups to consider in pursuing each of the respective pathways in re-establishing individual, family and community social and emotional wellbeing.

The figure depicts how each of the circles encompass the specific pathways to recovery related to each of three trauma themes. In combination they affirm that recognition of cultural strengths and facilitation of Indigenous ways of working, leadership, healing and empowerment is fundamental to promoting sustainable recovery. At the centre, each of the pathways of recovery come together to provide the holistic sense of health and social and emotional wellbeing for individual, family and community (Milroy et al., 2014).

It is important that groups, community and organisations have the means to achieve the things that matter to them. Working together to answer the following five questions should help you understand who you are as a group and what kind of governance they want to model to their children. Communities need to work out what their governance strengths and challenges are. One of the best places to start is with community members. The website includes a report outlining an Indigenous Governance Framework and an Indigenous Governance Toolkit. Questions worth considering include:

Considerations/Questions

  1. Who are we? What do we mean by community governance?

These questions are all about identifying which community, region or group you are part of, and who you are representing through your governance. Getting some agreement or resolution about these questions is a fundamental step, right from the start.

  1. Where did we come from? What’s the history of our governance?

This question is about looking back at the history of your governance and considering past practice and protocols and expectations.

  1. What have we got now? What is our current governance like?

This requires mapping all points of decision making in the community, considering issues of representation family and language groups and gender equity.

  1. What do we want? What future goals are we trying to accomplish for the future, how does this.

Answering this question requires thinking about how the community can build on strengths to address the key issues to reduce suicide. It will help develop a strategic vision for your governance.

  • What kind of governance are we trying to build for our group, our members, our children and the generations to come?
  • What do we hope will be different or better about our governance arrangements?
  • What do we want to stay the same?
  1. How do we get it? What is our strategy?

Answering this question is about considering how you can make your future vision happen.

  • What are our specific concerns and priorities?
  • What plans and actions should we develop and take?
  • Who will be responsible for doing it, and by when?
  • What resources do we need?
  • What are the risks involved and how can we deal with them?
  • How we are going to tell if we are making progress and getting the outcomes we want? (see the Indigenous governance toolkit website)

It is now widely recognised that the adverse effects of trauma on individual, family and community wellbeing is contagious and traumatic events effect whole communities because a community is a dynamic system that acts and responds as a whole ‘body’ (Sheehan et al., p.50). Several studies confirm the need to focus on healing to address trauma in order to improve Indigenous health, mental health and wellbeing outcomes and prevent suicide (Atkinson et al., 2014; Milroy et al., 2014; Powell 2014; Anderson, Baum & Bentley 2007). These studies emphasise the need for health promotion or education programs in suicide awareness and prevention to focus on healing and empowerment to help people gain a greater level of control over their lives and circumstances (Atkinson et al., 2014; Dudgeon et al., 2010; Healing Foundation 2014; Powell et al., 2014). What these different programs confirm is the importance of governments recognising Indigenous peoples as capable and efficacious individuals, as resilient, creative, adaptive with cultural knowledge, and wisdom to determine their own solutions.

There are a range of validated high-quality resources that are valuable for local mental health and social and emotional wellbeing workers addressing transgenerational trauma available on the CBPATSISP website and the Healing Foundation website:

  • ‘What’s up with My mob?’

Community members, practitioners and service providers engaging in suicide prevention will find a range of culturally responsive strategies, principles and practices to strengthen Indigenous mental health and social and emotional wellbeing in the following resources

Guide to Evaluating Organisations and Services

This section includes questions and considerations for assessing key evaluation criteria including the effectiveness and cultural responsiveness of programs and services, continuous quality improvement and for making decisions around funding and developing programs.

Organisations need to be resourced to develop their capacity for undertaking evaluation, continuous quality improvement (CQI) processes, (action research) through building an evidence base, fostering and, informing best practice in suicide prevention and early intervention within the public health sector.

This section includes evidence based, principles, processes and resources for Aboriginal controlled organisations, government and non-government organisations and services to use in planning, implementing and evaluating the effectiveness and culturally responsiveness of suicide prevention and early intervention initiatives.

All suicide prevention and early intervention initiatives, program and service delivery models implemented in Indigenous context need a sound theoretical base, appropriate conceptual frameworks, program logic and social indicators and performance measures to assess the effectiveness of programs, practices and services and delivery systems which are designed to simultaneously work with Indigenous children, families and communities;

  • Strengthen the capacity, leadership and empowerment
  • Support their cultural and social and emotional wellbeing
  • Address the social and economic determinants that are key risk factors

A clear understanding of the extent of issues
The population they are intending to ‘target’ or work with
A sound theory and logic for the program service or initiative they are evaluating

  • Step 1: Identify and prioritise issues for action
  • Step 2: Understand the issue
  • Step 3: Plan your program or service
  • Step 4: Implement and adapt your program or service
  • Step 5: Evaluate your program or service

What follows then is a set of questions or criteria to measure the key elements and principles identified as essential for partner organisations, program and service providers and policy personnel to determine whether programs and services have:

  • Increased the accessibility of services for people at risk of suicide or self-harm
  • Improved the cultural responsiveness and effectiveness of service delivery
  • Improved outcomes and influenced positive changes in the social determinants that contribute to poor mental health and wellbeing outcomes, high risk factors and suicidal behaviours, and suicide.

Broader questions also need to be answered to assess the efficacy, integrity and effectiveness of programs, services and strategies designed to reduce Indigenous suicide and address the associated risk and protective factors in complex and diverse circumstances. The key elements that are operationalised include:

  • Adopting a social determinants approach
  • Assessing initiatives to address racism
  • Embedding cultural responsiveness and cultural safety
  • Ensuring dual accountability and genuine partnership
  • Facilitating Indigenous governance and self-determination
  • Incorporating human rights
  • Facilitating empowered and socially sustainable communities
  • Strengthening protective factors
  • Supporting the healing of trauma, grief and loss
  • Establishing healthy communities
  • Developing/identifying resources for individual community healing and recovery
  • Developing/identifying workforce training and resources
  • Assessing program sustainability

Communities and organisations need to have the capacity to evaluate programs to enhance their understanding of ‘what works’, under what conditions and why in particular contexts (Baillie et al, 2013; O’Donoghue et al., 2014; Schierhout et al., 2013). Most organisations would benefit from having the support (by way of resources, knowledge and understandings) to undertake cycles of ongoing review and reflection. Continuous Quality Improvement (CQI) allows program and service providers to change things that are not working throughout their implementation, and to embed and document things that are working. Adopting a PAR approach, which is closely associated with CQI, enables organisations and services to engage in, and benefit from, action learning processes involving a process of critical reflection and a cycle of action, observation, reflection and planning to ensure program and service delivery is continuously improved. The value of measuring performance is in both monitoring progress towards achievement of agreed outcomes and in providing feedback on effective design and implementation of interventions. This continuous cycle of monitoring, review and refinement is illustrated in the Figure below extracted from the Australian Health Department website below.

There are a number of re-occurring themes around the support that organisations and programs need to ensure sustainable, culturally appropriate and effective service and program delivery – these themes reinforce the principles underpinning the Evaluation Framework. These principles include community negotiation, community engagement, and strengthening community capacity and community control and self-determination. These elements are critical to ensure organisational cultural competence and cultural safety in a deep and sustained way. The review of evaluations of existing programs such as the Yirriman and Mt Theo programs highlight many of these important effective characteristics of developing and implementing suicide prevention programs through Indigenous community controlled organisations.

Key questions any program or initiative needs to ask are:

  • How can the service, program or initiative achieve better outcomes?
  • What strategies and resources are required to ensure processes and practices and policies are effective and culturally appropriate processes to achieve improved outcomes in the short, medium and long term?
  • What are the most appropriate outcome measures to demonstrate the effectiveness of the service, program or initiative?

The One21seventy Health promotion CQI tools are an evidence-based resource package for use in Indigenous community settings. Health promotion tools and resources for the prevention of Indigenous suicide, self-harm and high risk behaviours can be successfully used by multidisciplinary teams with limited health promotion experience. The tools are accessible from here.

The CQI tools enable communities and service providers to:

  • Systematically describe and assess how well activities and projects align with good practice
  • Assess how well organisational systems are functioning to support health promotion
  • Plan how to improve systems that support good health promotion practice

The Evaluation Framework aims to ensure Indigenous cultural values, knowledge and terms of reference are respected, recognised and incorporated into all health, mental health, social and emotional wellbeing and other associated programs and services, including universal, targeted and indicated strategies to reduce suicide.

The use of PAR combined with Aboriginal-led and negotiated CQI in evaluating organisations and services ensures that Indigenous voices are given primacy and authority in determining the most appropriate social and performance indicators to measure improvements in processes and outcomes and restoring/maintaining Indigenous cultural protocols, practices, priorities and aspirations for the future.

There is growing evidence of the impacts of racism on health and social and emotional wellbeing outcomes and the subsequent association with poor mental health, suicide and self-harm. The experience of racism and social exclusion felt by Indigenous children at school, by Indigenous people in the workplace and when accessing health, education and social services has been shown to adversely impact on wellbeing outcomes and contribute to depression, despair and anxiety. Both the ATSISPEP consultations and the literature review findings confirm that Indigenous people continue to experience institutional racism (implicit or explicit, indirect or direct, unconscious or conscious) through the language, practices and processes of social policies and programs which impact upon every aspect of their lives. These findings highlight the need for health and mental health stakeholders to address racism, discrimination and marginalisation and enhance cultural competence in programs, services and initiatives to support suicide prevention and interventions. All communities and community controlled partner organisations have the right to expect state and national government organisations and services to address issues of racism and discrimination as part of their core business.

Institutional racism needs to be addressed at three levels:

  1. Individual (changing individual attitudes and behaviours)
  2. Intra-organisational (fostering a culturally inclusive culture, and positive relationships between staff and organisational policies and procedures)
  3. System-wide (including strategies to increase the awareness, commitment, willingness and resources to implement system level policies)

Some questions community organisations and service providers may consider include:

Questions

  • Are there recruitment and processes in place to encourage cultural affirmation?
  • Are there induction practices in place that promote a culturally safe work and service environment?
  • Is there a reconciliation action plan in place developed with local Indigenous groups?
  • Are there courses to support cultural learning in the workplace and in service provision?
  • Do staff understand what it means to enact the guiding principles?
  • Are the guiding principles to support social and emotional wellbeing of Indigenous individuals, families and communities consciously enacted?
  • Are there mechanisms and resources to embed anti-racism and cultural competence in individual and organisational practice?

Monitoring and evaluation of changes can occur through self-administered organisational audits by staff and community groups intended to benefit, such as online surveys, or interviews and focus groups. Evidence of the effectiveness of strategies might include improved awareness, knowledge, skills, capacity, networks, and reduced racial tension experienced by consumers, clients and community. Skilled, culturally aware leadership and clear positive communication about the importance of addressing race-based discrimination and its impacts is critical to support individual and organisational level change and reduce any resistance.

Approaches to implementing organisational accountability and development, diversity training and measures to increase positive inter-group contact can be found in the resources. Ideally, resources will need to be developed by and provided to workplaces on the basis of their specific needs and contexts of each organisation. Encouraging staff to participate in cultural competence and reflective learning courses is also important.

Cultural Competence – Transforming Policy, Services, Programs and Practice. Walker R, Schultz C & Sonn C. 2014, Chapter 12 in Dudgeon P, Milroy H & Working Together: Aboriginal and Torres Strait Islander Mental Health and Wellbeing Principles and Practice (2014). Canberra: Australian Government Department of the Prime Minister and Cabinet, 2nd edition. pp 195-220.

VicHealth – Building on Our Strengths:Framework identified five key factors in which organisations can play a key and important role in reducing race-based discrimination:

  1. Implementing organisational accountability/development
  2. Diversity training
  3. Resource development and provision
  4. Serving as sites for positive inter-group contact (Paradies et al., 2009; VicHealth, 2009)
  5. Serving as ‘role models’ in anti-discrimination and pro-diversity practice for other organisations (Trennery, Franklin & Paradies, 2012)

Ideas of dual accountability recognise that Indigenous organisations are accountable for the efficient use of funds in achieving program goals. At the same time, Indigenous people have fundamental rights to have access to funds and services which can contribute to their social and economic wellbeing and on their terms of reference. An extensive review by Dudgeon, Walker et al., (2014) found compelling evidence that both mainstream and Indigenous-specific programs and services that adhere to the Closing the Gap service-delivery principles of engagement, sustainability, access, integration and accountability are more effective than those that do not. The Closing the Gap service delivery principles are:

  • Indigenous engagement: Engagement with Indigenous men, women and children and communities should be central to the design and delivery of programs and services.
  • Sustainability: Programs and services should be directed and resourced over an adequate period of time to meet the COAG targets.
  • Access: Programs and services should be physically and culturally accessible to Indigenous people recognising the diversity of urban, regional and remote needs.
  • Integration: There should be collaboration between and within Governments at all levels and their agencies to effectively coordinate programs and services.
  • Accountability: Programs and services should have regular and transparent performance monitoring, review and evaluation.

In the spirit of genuine partnership, community controlled organisations are encouraged to ensure government and non-government organisations are held accountable to answer the following questions:

Questions

Do existing state and national government suicide prevention and intervention policies, programs and services have strategies to:

  • Protect and enhance the rights of Indigenous Australians to live and develop according to Indigenous terms of reference, values and needs?
  • Achieve equitable social and economic and wellbeing outcomes?

Such accountability questions need to be included in all evaluations. The Indigenous Evaluation Framework helps to inform government and organisations in responding to these questions.

Indigenous self-determination requires genuine partnerships between Indigenous people, governments and non-government agencies, together with a commitment to dual accountability. The notion of partnership along with self-determination is deeply embedded in human rights and is a key principle underlying the Evaluation Framework. Historically Indigenous peoples have been self-determining – defining their own being through control over all aspects of their lives, including ceremonies, spiritual practices, medicine, social relationships, management of land, law, and economic activities (Howitt 2013). There is substantial evidence to confirm that ‘being self-determining’ is a key protective factor in achieving suicide prevention outcomes. Having a sense of control over one’s situation is crucial to Indigenous social and emotional wellbeing.

The extent to which a program, service or initiative supports self-determination may be measured by evidence of how they support:

  • Indigenous cultural connectedness, language, spirituality and social and emotional wellbeing
  • Indigenous people contributing equally to decisions about practices and standards throughout all phases of the development, implementation and evaluation of programs and services

Questions

  • What processes and mechanisms are in place to ensure Indigenous people have been able to make a genuine contribution at all phases of the program/service development, implementation and evaluation?
  • Does the implementation of strategies, programs and services support Indigenous rights and values?
  • Does the program or initiative enable Indigenous people to have a sense of control over their lives?
  • Is this reflected in the indicators for the evaluation?
  • Does the program or activity support elements that contribute to an individual, family or community’s sense of being self-determining such cultural ceremonies and spiritual practices, social relationships, and economic activities?

Several studies emphasise the importance of incorporating human and Indigenous rights principles into evaluation and policy frameworks to improve Indigenous health and social and emotional wellbeing and achieve an equitable and sustainable future for Indigenous peoples. Recognition of human rights requires a greater commitment by governments to uphold Indigenous people’s individual rights to health, as well as their collective right to maintain and use their own health systems and practices in pursuit of their right to health. The recognition of human and Indigenous rights are higher order goals which are fundamental to the achievement of self-determination and equity, and ultimately to strengthening social and emotional wellbeing and reducing poor health outcomes including suicide.

Questions/Considerations

Importantly, a rights-based framework can also serve as a set of standards that can be used to assess the extent to which governments policies, services and programs:

  • Acknowledge Indigenous people as the traditional owners of the land and their connection with the land
  • Support and encourage connections and mutual respect between Indigenous and other Australians
  • Provide services based on understanding and respect for Indigenous culture
  • Develop and implement policies at the direction of the Indigenous peoples to redress past wrongs and support them to achieve the outcomes they desire
  • Recognise and work to address the continuing oppression, dispossession and disadvantage perpetrated on Indigenous people
  • Integrate Indigenous culture into all educational, cultural and ceremonial activities to strengthen relationships, understanding and recognition to provide beneficial outcomes to the whole community and reduce discrimination in the long term
  • Provide education and information about the Indigenous culture and history of the local area across the whole community to foster understanding and respect and maximise opportunities for Indigenous people to participate in local decisions

While the notion of capacity building is firmly embedded in government policy frameworks, there is a need for a shift in government thinking about program implementation and evaluation approaches that strengthen individual, family and community capacity.

Strengthening capacity is a two-way reciprocal process involving all stakeholders and especially those intended to benefit from the programs and services being planned, implemented and evaluated. All stakeholders need information to understand the different types and purposes of an evaluation, as well as familiarity with the concepts and language of community planning and evaluation. The CBPATSISP Clearinghouse website has links to a range of useful and culturally relevant resources and toolkits developed to assist evaluators in conducting empowering, participatory community evaluation included on government and community services websites on suicide prevention.

Guide to Evaluating Clinical Services, Protocol and Practices

This section outlines the key elements that need to be considered in providing clinically safe care, ensuring effective screening, diagnosis and treatment in a range of clinical settings to support Aboriginal and Torres Strait Islander people who have attempted suicide, experienced mental health issues or suicide ideation.

Recommendations from representative groups for clients/consumers, carers and their families have been integrated within each of the three guideline documents which can be downloaded from the Mental Health Standing Committee website at: http://www.health.gov.au/mhsc.

The discipline-specific guidelines that may apply include, but are not limited to, the following:

For Psychologists
From the Australian Psychological Society:
• Code of Ethics, Available from:
http://www.psychology.org.au/Assets/Files/NewCode160807WEB.pdf.

• Ethical and practice guidelines and procedures. Available for members only from:
http://www.psychology.org.au/about/ethics/

For Social Workers
From the Australian Association of Social Workers:
• Australian Social Work Education and Accreditation Standards (2012) Available from:
http://www.aasw.asn.au/document/item/100

• AASW Practice Standards for Mental Health Social Workers (2008) Available from:
http://www.aasw.asn.au/document/item/17

For Occupational Therapists
From the Australian Association of Occupational Therapists:
• Code of Ethics (revised 2001). Available from:
http://otaus.com.au/sitebuilder/about/knowledge/asset/files/1/codeofethics.pdf
• Australian Minimum Competency Standards for New Graduate Occupational Therapists 2010. From the Occupational Therapy Board of Australia.
• Code of conduct for registered health practitioners (2012). Available from:
http://www.occupationaltherapyboard.gov.au/Codes-Guidelines.aspx

For Psychiatrists
From the Royal Australian and New Zealand College of Psychiatrists
• position statements, clinical practice guidelines, ethical guidelines, Code of conduct and Code of ethics are available from:
http://www.ranzcp.org/Resources/StatementsGuidelines/Practice-Guidelines.aspx

For Nurses
From the Australian College of Mental Health Nurses:
• Standards of Practice for Australian Mental Health Nurses: 2010. Available from:
http://www.acmhn.org/images/stories/About-Us/standards_2010_web.pdf
From the Nursing and Midwifery Board of Australia:
• Code of Ethics for Nurses in Australia (2008)
• Code of Professional Conduct for Nurses in Australia (2008)
From the National Nursing Competency Standards for the Registered and Enrolled Nurses. Available from:
http://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/Codes-Guidelines.aspx

From the Congress of Aboriginal and Torres Strait Islander Nursing and Midwives (CATSINaM)
• National Summit on Cultural Safety in Nursing and Midwifery: Background paper for all mental health and health practitioner s/organisations link
From the National Aboriginal Community Controlled Health Organisation 2011,
• Creating the NACCHO Cultural Safety Training Standards and Assessment Process: a background paper, NACCHO, Canberra, http://www.naccho.org.au/promote-health/cultural-safety/
This background paper provides a comprehensive overview of concepts and meanings related to cultural safety and cultural safety training, including: cultural safety and respect as a human right, racism and cultural safety, good practice in cultural safety training. It also outlines a number of organisation strategies to achieve improved cultural safety at an institutional and individual level.

For General Practitioners
Australian Indigenous Doctors Association
• Position paper: Cultural Safety for Aboriginal and Torres Strait Islander doctors, medical students and patients, 2013, AIDA, Canberra.
AIDA’s suggest that strengthening cultural safety is a shared responsibility between educational institutions, national peak bodies and health services, as well as government and individual non-Indigenous medical practitioners

Resources

Ngarngadji! Listen/ understand! Improving Care for Aboriginal and Torres Strait Islander Patients (ICAP) Resource Kit

Evaluations need to assess the extent to which appropriate safeguards are in place to minimise the risks of harm. Some questions or issues to consider may include:

Questions/Considerations

  • Are there adequate safeguards in place in school and community settings to support community awareness and gatekeeping programs? For example is there ready access to culturally competent counsellors in schools or in local community?
  • Are there culturally responsive, family oriented mental health and wellbeing initiatives with clinical follow-up and referral for people identified with clinical depression or mental health disorder requiring medication?
  • Is there access to bereavement counsellors?
  • Have the prevention and postvention strategies and resources being adopted and promoted been formally evaluated and culturally validated?
  • Have issues of cultural safety /cultural security been addressed in program or service delivery?
  • Are strategies in place to consider issues of racism as outlined in the section on operationalising the principles in the CBPATSISP Evaluation Framework ?

The National Review of Mental Health Programs and Services (2014) recommended bolstering the availability and capacity of both community and culturally responsive primary health care services to support and respond to the mental health and social and emotional wellbeing needs of individuals, families and communities.

In addition, the findings from the Australian Institute of Health and Wellbeing Review (AIHW, 2013) confirm that clinical approaches that have proved to be effective in reducing suicide rates include:

  • Community programs that focus on the social, emotional, cultural and spiritual underpinnings of community wellbeing
  • Culturally responsive brief interventions comprising motivational care planning

Programs and services that are not culturally competent and do not have a high level of Indigenous ownership and community support are unlikely to be effective in responding to the diverse and particular needs of Indigenous people in addressing suicide (AIHW, 2013).