Centrality of Culture
Most government frameworks about Aboriginal and Torres Strait Islander peoples acknowledge the centrality of culture in Indigenous health and wellbeing. The critical importance of culture underpinned the Cultural Respect Framework 2016-2026 for Aboriginal Health and National Strategic Framework for Aboriginal and Torres Strait Islander Peoples’ Mental Health Social and Emotional Wellbeing Framework 2017-2023.
However, a review of these frameworks revealed that key recommendations to support human rights and self-determination in health and mental health and wellbeing services to prevent suicide and poor mental health outcomes, trauma and grief and family dysfunction have not been implemented and resourced during the past decade.
This section outlines the conceptual models for conducting evaluation and includes the importance and relevance of understanding Indigenous contexts. The conceptual model is context specific-it requires taking account of and assessing the overall context in which suicide and self-harm is (or is not) occurring and then setting a goal/s to reduce through direct or early intervention (for example: reducing suicide, improving child and maternal health in Indigenous communities strengthening the capacity of children and families, or connecting young people to country).
The ecological approach can meet a community’s distinctive needs through the development, implementation and evaluation of the effectiveness of intervention projects targeted at critical points across the life-course. This approach enables Indigenous communities to draw on their local terms of reference to map a community’s readiness for behaviour change, level of capacity and infrastructure, the cultural interactions within and between groups, apply a ‘theory of change’ logic based on local knowledge and understanding throughout the planning, implementation and evaluation processes, and develop tools to measure the effectiveness and cultural responsiveness of selected programs and services.
An analysis of over 2000 community consultations to obtain Indigenous community perspectives on Indigenous suicide prevention reinforce the need for ‘community-focused, holistic and integrated approaches to suicide prevention, with intervention strategies that reduce the likelihood of suicide and related problems over the lifespan’ (Commonwealth Department of Health and Aging, 2013). This is consistent with the extensive and recent national and international research evidence, which now shows how environmental and biological factors influence brain development in early life and with far greater effects on adult outcomes in physical and mental health and social and emotional wellbeing than previously understood. This has significant implications for Indigenous individuals and families given the interplay of distinctive risk and protective factors associated within the broader social, cultural historical contexts which differ from those experienced by the wider population. It is critical to understand how these social and environmental determinants impact upon Indigenous social and emotional wellbeing, together with the critical and urgent need for greater emphasis on investment in initiatives to strengthen the capacity of communities to prevent psychosocial and behavioural problems in early childhood and adolescence.
The ATSIPSEP research called for greater investment in ‘upstream’ preventive policies and services which build community, family and individual resilience to reduce suicide and self-harm. The NATSISPS called for a strategic alignment of policies and targeted investment in early prevention in health, family and children’s services, education and mental health at key points across the lifespan. The evidence shows that strategies that support children’s social and emotional learning, individual identity, and promote resilience and self-determination can reduce vulnerability to future adverse outcomes.
A key aim of ATSISPEP was to develop an Evaluation Framework to provide an evidence base of ‘what works’ to ensure programs and services are most effective in reducing the high rates of Indigenous suicide and promoting social and emotional wellbeing to support the implementation of the now expired National Aboriginal and Torres Strait Islander Suicide Prevention Strategy (NATSISPS). The development process involved a series of targeted Roundtables, an extensive literature review, the collation and analysis of findings from all previous community consultations about suicide, the identification of suicide patterns and trends over time and between regions, and the development and validation of an Indigenous Evaluation Framework to identify best practice and suicide prevention programs, services and resources. The ATSISPEP Evaluation Framework also incorporated the principles and key findings and the key themes, principles and indicators of major national and state community consultation report findings.
The ATSISPEP Evaluation Framework provided a tool to support the effective implementation, monitoring and evaluation of the recommendations of NATSISPS. The Evaluation Framework was applied to several existing suicide prevention programs and initiatives which align with the Action Areas and outcomes in NATSISPS, which is now subsumed within the Fifth National Mental Health and Suicide Prevention Plan. The findings were presented in a report, A Summary of Promising Programs, Services and Resources in Aboriginal Suicide Prevention and Postvention which outlines a range of programs, services and resources of ‘what works’ as well as promising practice. The evidence base of ‘what works’ informed recommendations for systemic change to reduce suicide that are included in the Solutions that Work Report (Dudgeon et al., 2016). These programs and resources will be revisited and permission sought to review their current status in accordance with the CBPATSISP Evaluation Framework principles and included on the CBPATSISP Clearinghouse as exemplars of best practice.
The CBPATSISP aims are as follows:
1. Identifying the need for and facilitating, innovative new research (including evaluations of un-evaluated activity) to support the further identification of Indigenous best practice.
2. Assessing best practice by Primary Health Networks (PHNs) in planning and commissioning Indigenous suicide prevention activities.
3. Working to translate best practice for application in Indigenous communities, community organisations, and by PHNs. This includes developing accessible and appropriate guidance and resource materials.
4. Developing an Indigenous-specific adaptation of the systems approach (e.g. European Alliance Against Depression model) to suicide prevention, based on identified best practice and aligned with the current overarching approach.
5. Proactively promoting and disseminating best practice research to ensure accessibility for all stakeholders through a Clearinghouse developed in consultation with and technical support from the Australian Indigenous HealthInfoNet and National Conferences and the creation of a responsive education/guidance program tailored to stakeholder needs. This includes:
- A best practice in Indigenous suicide prevention education program delivered to PHNs with follow up over the life of the program
- A complementary education best practice program and resources specifically for Indigenous communities and organisations to help them work effectively with PHNs
- A best practice email advisory service for communities, community organisations and PHNs
- A key aim of CBPATSISP Clearinghouse is to further refine the ATSISPEP Evaluation Framework which was developed to measure the effectiveness and cultural appropriateness of suicide prevention initiatives
The CBPATSISP Evaluation Framework comprises: evaluation methods and methodologies, a program logic, theory of change, and social indicators and performance indicators to measure processes and results in achieving goals and aims. Bringing together these elements within this Evaluation Framework addresses many of the gaps, inconsistencies and misguided assumptions identified in existing evaluations of program and services. Operationalising each of these elements within the CBPATSIP Evaluation Framework presents an opportunity to forge a new direction in evaluation to demonstrate social impact that is both far-reaching and local in addressing the goals of various reports such as the ATSISPEP Report. The ability to demonstrate short and medium outcomes with a high likelihood of reaching long term gains over time has potential in turn to result in government, industry, philanthropic groups and communities themselves investing in programs and services that are effective, cost effective and culturally appropriate, robust and self-sustaining.
A key aim of the Centre for Best Practice in Aboriginal and Torres Strait Islander Suicide Prevention is to further progress the implementation and widespread uptake of governments and other key stakeholders utilising the CBPATSISP Evaluation Framework in evaluating programs and services. The CBPATSISP role in providing funders with tools to assess proposals, and to invite programs and services providers communities and clinicians to evaluate the effectiveness and cultural responsiveness.
Drawing on the ATSISPEP work, the CBPATSISP Evaluation Framework outlines:
- A methodology to conduct culturally appropriate and comprehensive assessments of programs, services and interventions with regard to rigour, program impact, and community and stakeholder responses to the suicide prevention programs
- Culturally relevant indicators to measure the effectiveness of programs and services addressing social and economic determinants on Indigenous social and emotional wellbeing outcomes
- The steps and key considerations for communities and organisations to plan, implement and evaluate their own community level suicide prevention activities
The most effective suicide prevention involves both whole of community and whole of government approaches. A whole of government approach encourages and facilitates the implementation of comprehensive suicide prevention programs through the development of program linkages and partnerships between government service providers, Indigenous organisations and peak bodies in the delivery of Indigenous programs and through the specification of the goals in program evaluation criteria.
From a government and Indigenous leadership perspective, positive change requires system level change based on evidence of ‘what works’ to promote effective and sustainable suicide prevention. The findings of ATSISPEP literature review confirm that to effect system level change to reduce suicide and self-harm will require:
- Genuine engagement with Aboriginal people, communities and organisations to promote sustainable, culturally responsive and effective services and programs
- Greater interagency understanding of, and commitment and capacity to address, the variability of human capability and community readiness in Indigenous populations as a consequence of risk factors such as poverty, illness, incarceration, low self-esteem, trans-generational trauma and its relationship to suicide and self-harm
- Greater support for families and communities by directly strengthening the protective factors and implementing programs that contribute to pathways to recovery
- Resources and strategies that strengthen the commitment, understanding and capacity among mental health, health, education, justice, child protection and social services and their staff
- Strategies and policies that are designed to reform and transform systemic, organisational and individual racism and discrimination and contribute to the development and embedding of cultural capabilities and competence across all relevant services and professions (Calma & Dudgeon, 2009; ATSISPEP, 2015; Walker, 2010, Walker et al., 2014)
- Workforce development, Indigenous health curriculum and Indigenous governance and leadership
- Primary and tertiary services that are culturally responsive, flexible increase access and integration and partnerships
The CBPATSISP Evaluation Framework provides an overarching mechanism to demonstrate the extent to which these elements are implemented, monitored and evaluated by the relevant government and non-government agencies to successfully achieve goals of the Fifth National Mental Health and Suicide Prevention Plan, relevant jurisdictional implementation plans and the renewed Social and Emotional Wellbeing Framework 2017-2023.
This requires State and Commonwealth Governments to genuinely engage with Indigenous Australians to develop local, culturally responsive approaches to identify and respond to individuals, families and groups most at risk within Indigenous communities, and support the evaluation of the cultural responsiveness and effectiveness of programs initially funded through the National Aboriginal and Torres Strait Islander Suicide Prevention Strategy, and now Primary Health Networks (PHNs).
The development of the ATSISPEP Evaluation Framework was informed by a comprehensive national and international literature review, Indigenous community consultations between 2009 and 2015, the ATSISPEP Roundtable consultation findings, advice and direction, Indigenous expert leadership group, and the National Aboriginal and Torres Strait Islander Mental Health Indigenous Leadership (NATSIHMIL). The CBPATSISP Evaluation Framework served two main purposes for different audiences:
- Firstly, as a resource guide to support community, planning, ownership and delivery of best practice suicide prevention programs to reduce suicide and suicidal behaviours in specific communities and for specific population groups.
- Secondly, it provided a set of principles and standards to review existing programs, services, training programs and resources (that have been previously evaluated and those that have been identified as promising practice but not yet evaluated) to assess the extent to which they are both effective and culturally appropriate.
These have been incorporated into the CBPATSISP Evaluation Framework to provide practical, ongoing evaluation approaches, methodologies and resources for program stakeholders that include:
- Methods and performance indicators to measure the effectiveness and cultural responsiveness of Indigenous suicide prevention programs, services and strategies which presents a new way forward
- Principles, standards and social wellbeing indicators developed on basis of the extensive community consultations and expert advice obtained through ATSISPEP National Advisory Group
- A set of guidelines to integrate and implement Indigenous determined, culturally relevant, evidenced-based principles and indicators within programs, services and initiatives which aim to address suicide prevention
Furthermore, the CBPATSISP Evaluation Framework reflects Indigenous Australian peoples’ holistic view of health, mental health, physical, cultural and spiritual health. It takes into account the complex issues and determinants contributing to Indigenous suicide at individual, family and community levels and across diverse geographic contexts. Importantly, the CBPATSISP Evaluation Framework is underpinned by Indigenous understandings of the causes and solutions to suicide as well as evidence based national and international best practice in Indigenous research, evaluation and suicide prevention.
The CBPATSISP Evaluation Framework comprises several critical components to evaluate the key outcomes related to community based programs, universal, indicated and targeted programs, workforce initiatives, clinical services and organisations and agencies delivery of suicide prevention strategies and resources for different stakeholders. In summary, the CBPATSISP Evaluation Framework provides a mechanism to monitor, evaluate and report on:
- The effectiveness and cultural responsiveness of suicide prevention programs and services
- Community, organisational and service level evaluations of outcomes of programs and services
- Complex multilevel and integrated services strategies
The CBPATSISP Evaluation Framework builds on the ATSISPEP project foundations to ensure that assessments of best practice, promising practice, what works and what needs to happen are informed by Indigenous knowledge, perspectives and priorities.
Thus, evaluation methods and methodologies to assess the effectiveness and cultural appropriateness of prevention programs are designed to address complex issues, including social determinants and inequities contributing to Indigenous suicide at individual, family and community levels and across diverse geographic contexts. The CBPATSISP Evaluation Framework builds on the ATSISPEP project foundations to ensure that assessments of best practice, promising practice, what works and what needs to happen are informed by Indigenous knowledge, perspectives and priorities.
Thus, evaluation methods and methodologies to assess the effectiveness and cultural appropriateness of prevention programs are designed to address complex issues, including social determinants and inequities contributing to Indigenous suicide at individual, family and community levels and across diverse geographic contexts.
The CBPATSISP Evaluation Framework provides the basis to:
- Assess the extent to which key principles and processes to enhance Indigenous self-determination, such as equal partnership, dual accountability and negotiation, are enacted in the development, implementation and evaluations of suicide prevention initiatives and interventions
- Assess how various programs which aim to address risk factors strongly associated with suicide and self-harm in urban, rural and remote areas effect or contribute to wellbeing outcomes
- Assess the extent to which programs, policies and practices strengthen community and build capacity (in both Indigenous and broader societal contexts)
- Serve as a set of criteria by which to assess/critique the appropriateness and usefulness of existing social and performance indicators used by government and non-government agencies to measure the effectiveness of services, programs and interventions intended to reduce suicide and self-harm in diverse community and geographic contexts
- Provide an inventory of different context-specific best practice examples to enhance program delivery in similar settings to more accurately inform policy decisions and resource allocations
- Establish indicators to measure the extent to which Indigenous goals, terms of reference, equitable processes and outcomes are achieved
Establishing strong families and communities:
- Are there positive programs that support families and extended kin to provide nurturing activities for infants and children?
- Are there programs that address domestic violence in ways that help overcome negative behaviours and poor long term outcomes, alternative mediation strategies?
- Are there community wide strategies to support healing, empowerment and leadership?
Developing and integrating ‘Two-way’ education:
- Does the school include curriculum teaching activities that acknowledge and support Aboriginal culture?
- Is there a language program being conducted by local Aboriginal people?
- Does the school actively promote cultural reconciliation, celebrate NAIDOC week?
Activities for youth:
- Are there program activities in the community for engaging young people to help them to develop positive social interactions with their peers and discover alternative activities to avoid risky behaviours?
- Does the community have programs in place for disengaged young people? (see Appendix 2 for resources for young people)
Establishing culturally responsive employment or meaningful activities:
- Are there culturally-appropriate employment pathways and work opportunities to give meaning and purpose to people’s lives?
For the summary of promising practice programs and initiatives, the fact sheets and Clearing House Paper outlines a range of programs, practices and the types of programs, see Appendix 4, p. 85 of the ATSISPEP Final Report, 2016.
Government and industry organisations generally specify a range of Key Performance Indicators to assess the effectiveness, (cost effectiveness), efficacy, efficiency and cultural responsiveness and safety of programs they fund. Assessing the effectiveness of implementing suicide prevention programs and strategies is a rigorous approach to ensure that what a community or organisation is doing or plans to do actually works.
In Aboriginal and Torres Strait Islander contexts it requires the following steps or considerations:
- Identifying which strategies will be most likely to reduce self-harm and death from suicidal behaviour
- Determining the potential effects of cultural, social, historic, legal, ethical, and economic factors on those strategies
- Determining the most effective, culturally responsive and culturally safe methods for implementing those strategies
- Assessing the effectiveness of the strategy in achieving its intended outcomes as it is developed and implemented
The CBPATSISP Evaluation Framework outlines steps to conduct formative, ongoing, continuous and summative stages of evaluation, process and outcome approaches throughout the development, implementation and evaluation of the project. It includes questions and mechanisms to assess the cultural responsiveness and effectiveness of initiatives at each of these phases. It includes a set of questions to be asked in determining the relevance of a proposed program, service or initiative and their alignment with the goals of relevant government strategies. It also includes a set of questions to determine the extent to which the project or activity can provide evidence that the longer term benefits of the outcomes are likely to outweigh the costs.
The higher the level of risk of the target population, the more intensive the prevention effort is required and the earlier it needs to begin. The following considerations are crucial for policymakers and program funders in determining both the effectiveness of suicide prevention strategies and policy and funding priorities when making decisions regarding:
- Potential to reduce or avoid self-injury or death
- Social, legal, and ethical and economic impact
- Best methods to ensure cultural safety
Funding agencies, service providers and communities also need to know ‘what works’ in developing/adapting and implementing suicide prevention programs and services to address community identified needs and priorities outlined in the Solutions that Work (2015). This requires being able to identify the features of effective, culturally responsive programs, services and initiatives in order to achieve successful outcomes.
The most important things funding providers need to consider in the first instance include:
- The delivery approach of the program or initiative
- The formation of partnerships
- Integration and co-ordination of interagency and inter-sectoral services
- Capacity building (of staff and of groups /communities intended to benefit)
- Targeting approaches (which population group/s is/are involved? and whether universal, indicated or targeted approaches to suicide prevention are required)
- The use and quality of evidence and research in establishing the rationale for the proposed initiative
- What works best for whom, why and when?
- Does the service, program or initiative help achieve outcomes in other relevant and related program areas? (eg. Are there links with the drug and alcohol strategy and/or through interagency and/or other government programs?)
- Does the service, program or initiative include strategies to address the local contextual issues that may hinder the effective implementation of key Aboriginal and Torres Strait Islander priorities in the Fifth Mental Health and Suicide Prevention Plan in achieving its objectives and outcomes?
- Does the program or activity include evidenced based ‘what works’ elements that are likely to make a positive difference as outlined in the ATSISPEP Solutions?
That Work Report
- Is there a sound theory or evidence base (which addresses social determinants and casual pathways) to help explain why some initiatives are most likely to work?
- To what extent does the program or activity take account of the key principles underpinning the CBPATSISP Evaluation Framework and key elements identified in community consultations (see table 1)?
- Is there an evidence-based rationale to demonstrate how the goals and objectives of the project/activity are designed to address identified risk and protective factors for a specified group, local community or whole population? And if so,
- Is there a sound statistical data and evidence base to inform the selection of the project, program and activity?
- Is there evidence of strong community consultation and leadership in determining the issues and priority area being addressed and in implementing the program?
Effectiveness is the real world impact of the prevention activity – that is the extent to which the level of care, intervention or action achieves the desired outcome/s. It is the efficacy of a strategy taking into account the specific challenges, constraints and opportunities within the specific contexts and communities. For example, a theoretically sound program may have to address a range of local issues that act as a barrier to participation.
The effectiveness of programs, services and initiatives is generally best understood and assessed and demonstrated in the community by involving community people who were intended to benefit. In assessing the effectiveness of a program, service or initiative it is important to ask;
- How well does the program, service or initiative work in the real world?
- If it is a program, service or initiative or set of initiatives does it have a sound theory of change and program logic?
- What is known about the local context?
- Have steps been taken to address the barriers and support the opportunities and strengths?
- Have all the right people in the community been included?
- Are all the relevant community organisations involved?
At this point in preparing any funding submission intended to address suicide prevention or postvention at the community level it is necessary to show how all of the issues identified in the community mapping are going to be addressed.
It is also the point where the responses to the questions generated around operationalising the principles and goals underpinning the CBPATSISP Evaluation Framework become crucial in determining the breadth of strategies necessary to strengthen and address individual, family and community level issues as well as consider system level issues.
Evaluations need to assess the extent to which appropriate safeguards are in place to minimise the risks of harm. Some questions or issues to consider may include:
- Are there adequate safeguards in place in school and community settings to support community awareness and gatekeeping programs? For example is there ready access to culturally competent counsellors in schools or in local community?
- Are there culturally responsive, family oriented mental health and wellbeing initiatives with clinical follow-up and referral for people identified with clinical depression or mental health disorder requiring medication?
- Is there access to bereavement counsellors?
- Have the prevention and postvention strategies and resources being adopted and promoted been formally evaluated and culturally validated?
- Have issues of cultural safety /cultural security been addressed in program or service delivery?
- Are strategies in place to consider issues of racism as outlined in the section on operationalising the principles in the CBPATSISP Evaluation Framework ?
The following considerations are crucial in assessing the extent to which the goals, aims, purpose and outcomes of suicide prevention and early intervention programs, services and initiatives for the specific group intended to benefit are cultural appropriate. The considerations are equally important for mainstream, universal programs (including Indigenous community) as they are for indicated and targeted programs and services.
Questions those undertaking the evaluation (including in the formative phase of development) need to ask to what extent:
- Does this program/initiative/service focus on all or some aspects of the physical, spiritual, cultural, emotional and social wellbeing of an individual, their family and community?
- Does this program/initiative/service focus on all or some aspects of the physical, spiritual, cultural, emotional and social wellbeing of an individual within specific risk groups (ie LGBTQI)?
- Do workforce initiatives encourage and resource frontline workers and practitioners to focus on the physical, spiritual, cultural, emotional and social wellbeing of the individual, family and community?
- Does the program/initiative/service or workforce initiative/resource recognise trans-generational trauma and align with the social and emotional wellbeing principles and principles of empowerment?
- Does the program/initiative/service aim to strengthen cultural values and commitments, family and kinship systems of care, and Indigenous control and responsibility as an intrinsic aspect of healing and facilitating cultural, social and emotional wellbeing?
- Does the service/program or workforce initiative acknowledge and work in partnership with the Aboriginal and/or Torres Strait Islander community-controlled sector and facilitate Indigenous people’s right to determine the types of services they receive?
- Does the service or program or workforce initiative have a statement of commitment and strategies to address racial discrimination?
- Does the program/initiative support human rights and social justice principles?
- Does the program/initiative support Indigenous organisations cultural governance/self determination?
Program and service providers or external evaluators undertaking the evaluation (including in the formative phase of development) need to work with Aboriginal and Torres Strait Islander people (as co-designers) assess the quality of the program plan including the evaluation to build in key principles and culturally appropriate methods and methodologies from the onset.
A range of risk factors can impact on the sustainability of a program or initiative regardless of its effectiveness, including the short-term nature of most funding programs, difficulties in obtaining and retaining suitable personnel and poor participation rates due to lack of transport and family constraints, lack of community involvement in the program development. Many initiatives remain as pilots and are not adequately evaluated.
Community ownership of, and genuine partnership and contribution throughout all stages of the development and implementation process are fundamental to the program and can help offset these issues. In order to assess the extent to which community controlled or government or non-government organisations delivering a program or initiative actively took steps to achieve sustainability it is important to consider the following questions:
- Does the project or initiative ensure a culturally responsive, localised and stable Indigenous control and governance model?
- Is there a long-term vision, leadership and support for the specific initiative, project or activity within a community, region or state?
- Has there been broad and ongoing Indigenous community consultation and strategic planning?
- Is there evidence of genuine engagement with community groups and the community controlled health sector as well as with relevant stakeholders and institutions?
- Is there evidence of collaborative activities being undertaken to support building, linking and leveraging relevant relationships and programs with partner organisations to strengthen sustainability?
- Have funders contributed to constructive and preventive ‘upstream’ investments to achieve positive goals that contribute to Indigenous wellbeing, promote resilience and reduce suicide?
- Have program funders and service providers a commitment to incorporating an appropriate set of indicators to evaluate the effectiveness of a program or initiatives’ short, medium and long term outcomes?