Evaluation

Clark, Y., Augoustinos, M., & Malin, M. (2017). Evaluation of the preventing lateral violence workshop in Adelaide, South Australia: Phase 1 survey responsesJournal of Indigenous Wellbeing: Te Mauri-Pimatisiwin, 2(3), 41-53.

Education and awareness workshops are important resources to the Indigenous community in Australia to tackle difficult issues and situations within communities. It is equally important that such workshops are appropriately evaluated to ensure that they are relevant, refined and continued within communities. This paper focuses on an evaluation of the one-day Preventing Lateral Violence workshop. A sample of six workshops were conducted predominately in Adelaide in 2014 using pre, post and three months post workshop measures to determine the impact on participants. Phase 1 measured participant changes in four areas which were awareness, understanding, knowledge as well as prevention strategies of lateral violence. Analysis of the quantitative data revealed the workshops’ success as an intervention tool both at completion and three months after the workshops. The results indicated that participants understood and resonated with the terminology and the impact on self and others, recognized their experiences of lateral violence, and had an awareness of and developed strategies to combat lateral violence. The qualitative response to open-ended questions suggested that various strategies of education and the challenging of behaviours were in place even after three months. The implications of these findings are discussed.

Clark, Y., Augoustinos, M., & Malin, M. (2017). Evaluation of the preventing lateral violence workshop in Adelaide, South Australia: Phase 2 qualitative aspectsJournal of Indigenous Wellbeing: Te Mauri-Pimatisiwin, 2(3), 54-66.

Lateral violence describes how members of oppressed groups direct their dissatisfaction toward each other. This inward deflection has been associated with Indigenous communities around the world and has shown to be destructive. The focus of this research concerns Aboriginal Strait Islander people in Adelaide, South Australia, as part of an evaluation of the preventing lateral violence workshops. The overall evaluation comprised both quantitative and qualitative components. This article reports on qualitative data, from interviews with seven Aboriginal participants, post workshop. These interviews, examined their ways of dealing with and strategizing to prevent lateral violence in various contexts as well as suggestions for improvements to the workshops. There were several interpretive themes that emerged from these interviews. This paper reports on the three main themes: improvements to workshops; participant support needs and their strategies to prevent lateral violence in their contexts. The information complemented and provided a deeper understanding of the phase one evaluation. It is hoped that such evaluation provides robust evidence for workshops to improve and be maintained as a useful resource for the Aboriginal and Torres Strait Islander communities to prevent lateral violence.

Guenther, J., & Galbraith, M. (2014) Learning from evaluations of school-family strengthening programs: lessons for all

For more than 10 years, a program called Families And Schools Together (FAST) has been run in schools across the Northern Territory. These programs have always had an evaluation component built in. However, over time, the evaluations have changed. Initially, they were summative, built around a quantitative psychometric tool (with a positivist research paradigm). The intent of the summative evaluation was to demonstrate the effectiveness of the program. However, as the program was rolled out in remote contexts, the need for adaptation was recognised. Changes were made but it was soon recognised that other outcomes, not captured or explored in the methodology, were emerging. After six years of working together, the evaluator and the program manager felt that it was time to explore effectiveness in a different way with a more qualitative evaluation process (based on naturalistic and pragmatic paradigms). The purpose of this paper is to share learnings from this experience with other program managers and evaluators. The presentation will be an opportunity for participants to engage in a discussion about monitoring and evaluation from a program evaluation perspective, particularly taking account of the complexities of the northern Australian context. This paper explores the six year learning journey through evaluation that the evaluator and program manager have undertaken. It describes tensions between the need for reliable and generalizable objective quantitative data and the need for authentic and credible data based on participant experience.

Maughan, C. (2012)  Monitoring and evaluating social impacts in Australia, Ninti One Limited

Abstract: Monitoring and evaluation (M&E) allows people to learn from past experiences, improve service delivery, plan and allocate resources and demonstrate results as part of accountability to stakeholders. M&E also assists in keeping projects on track, providing a basis for reassessing priorities and creating an evidence base for current and future projects There is a growing interest in the measurement of social impact across the business, government and nonprofit sectors. In recognising the role that non-profit organisations play in ‘enhancing the economic, social, cultural and environmental wellbeing of society’, the Australian Government has recently focused on improving the measurements of social impact This report describes the main monitoring and evaluating frameworks and methods used in Australia which include some assessment of social impact. Each of the following are discussed in terms of an overview of how it works (in terms of M&E), the benefits and limitations, examples of organisations using the technique and where to find further resources.

Murrup-Stewart, C., Searle, A.K., Jobson, L., Adams, K. (2019). Aboriginal perceptions of social and emotional wellbeing programs: a systematic review of literature assessing social and emotional wellbeing programs for Aboriginal and Torres Strait Islander Australians perspectivesAustralian Psychologist54(3), 171-186.

Australian Aboriginal and Torres Strait Islander communities have been targets of social and emotional wellbeing programs for many years. However, the few health‐care programs and services that are evaluated rarely provide insight into the participants’ perspectives of program success or failure. This systematic review assessed 33 social and emotional wellbeing programs across Australia to better understand what Aboriginal community members think about the programs and how they could be improved. Results highlighted the interesting and valuable insights provided by Aboriginal participants, including what kinds of program activities and approaches are most suitable, what program characteristics are successful or desired, and their experiences of wellbeing change before and after program participation. They likewise voiced opinions about poorly received programs, culturally inappropriate services and negative experiences. This review highlighted how health and wellbeing programs must better engage Aboriginal and Torres Strait Islander clients to ensure that services are culturally safe, holistic, integrate appropriate staffing, include culturally relevant activities and value patient/participant experiences. These findings have significant implications for the health and wellbeing sector; specifically, research, policy, program design and implementation, evaluation methods, and self‐determination.

Rogers, Alison, Harrison, Nea, Puruntatameri, Therese, Puruntatameri, Alberta, Meredith, Joan, Dunne, Rachel 2018, Participatory evaluation is the sea eagle looking “long way wide eyed” https://journals.sagepub.com/doi/abs/10.1177/1035719X18778712

 Abstract: Participatory evaluation can be embedded in programs to support good governance and facilitate informed decision making in Aboriginal communities in remote and urban contexts. An Aboriginal Elder from the Tiwi Islands in the Northern Territory of Australia described participatory evaluation as a sea eagle looking “long way wide eyed.” The metaphor refers to the long-term and broad approach undertaken when a complex community development program used participatory processes to build evaluation capacity and solve problems. The evaluation approach ensured the program was inclusive, responsive, empowering, and resulted in direct benefits for the communities. This article addresses the lack of literature on applying developmental and empowerment evaluation approaches in practice by describing the methods, tools, and use of evaluation findings. The value of participating for the community members and partner organizations is shared and the benefits and implications for participants and the evaluator are discussed. The authors hope this article inspires practitioners and evaluators to consider participatory ways of working with communities to support community directed action and social change.

Evaluation of Gathering Places: Inclusion, connection and empowerment – Victorian Government

In 2015 the Indigenous Health Equity Unit (IHEU) at The University of Melbourne was commissioned to undertake an evaluation of 13 identified Aboriginal and Torres Strait Islander gathering places across the state. The focus of the evaluation was to explore the extent to which gathering places in Victoria impact on the health and wellbeing of Aboriginal and Torres Strait Islander communities across the state.

Fisher, S. (2012) Monitoring and evaluation methodologies for remote settings: A literature review conducted in 2010 Report to the Office of Indigenous Policy Coordination, Department of Families, Housing, Community Services and Indigenous Affairs Working Paper  http://www.nintione.com.au/resource/NW001_MonitoringAndEvaluationMethodologiesForRemoteSettings.pdf

This report was undertaken by Ninti-One and provides useful material for staff working on new initiatives in monitoring and evaluation. Drawing on a review of the literature this report provide an overall analysis and recommendations regarding the conduct of monitoring and evaluation in Indigenous remote communities.

Hand, K. (2018). Gift of Gallang: Evaluation. Brisbane, QLD: Mission Australia.

In 2018 the Gift of Gallang was implemented in a local primary school in Inala for the first time. This represented the culmination of a three-year journey of ongoing community engagement and collaboration, which included the formation of the Committee of Hope and the development of community engagement activities such as cultural nights. The need for creating hope, resilience and wellbeing in the community of Inala was evident after a cluster of youth suicides occurred in 2016. Creating a community where children had a strong sense of culture and connection framed the Gift of Gallang activities with the long-term aim of reducing risk of suicide for Aboriginal and Torres Strait Islander children and young people. This work was supported by partnership between Mission Australia, and several other key stakeholders, including Inala Wangarra, local Elders, other Aboriginal Torres Strait Islander community and health organisations, and a local primary school. Mission Australia also funded evaluation of the processes involved throughout this work in order to document key learnings. Data was collected from members of the Committee of Hope, facilitators, educators, parents and carers. Before data collection Ethics approval was obtained through the Mission Australia research team as well as the Queensland Department of Education for data relating to school staff, children and families.

Skerrett, D.M., Gibson, M., Darwin, L., Lewis, S., Rallah, R., De Leo, D. (2017) Evaluating the  “United Health Education and Learning Program” (UHELP) programClosing the Gap in Aboriginal and Torres Strait Islander Youth Suicide: A Social–Emotional Wellbeing Service Innovation Project. Australian Pyschological Society, Vol 53, Issue 1. doi.org/10.1111/ap.12277

The suicide rate for Queensland’s Aboriginal and Torres Strait Islander young people is over four times that of their non‐Indigenous counterparts, with Aboriginal and Torres Strait Islander children (under 15) dying by suicide at 12 times the non‐Indigenous rate. There is a need for interventions that are culturally validated and community‐endorsed. The aim of this article is to describe the design and implementation of a group‐based intervention, as well to report the results of the various qualitative and quantitative measures.