The potential value of the Realist Evaluation Method in programme impact evaluations in South Africa

[摘要] ENGLISH SUMMARY : In the context of the development of a coherent country-driven monitoring and evaluation (M&E) system in the South African public sector underpinned by the policy outlining the Government-Wide Monitoring and Evaluation System (GWMES), this study explored the methodological approachesapplied in impact evaluations. With the field of evaluation theory still booming internationally, there are no prescribed guidelines as to which impact evaluation method, employed in a specific context and under which conditions, will render the most useful findings for policy-makers. Therefore,clarifying a suitable methodological approach for impact evaluation of social programmes is criticalin South Africa. Towards these aims, this study explored appropriate methodological approaches ininforming better programme impact evaluations through the exploration of the potential value of Realist Evaluation Method (REM) on impact evaluations within the South African public sector. The research had a three-pronged approach. First, a comprehensive literature review was conducted to gain an in-depth understanding of the REM through a detailed review and analysis of the related literature, as well as to assess the current trends in research and application of REM approach inconducting impact evaluations. Secondly, case study micro-analyses were completed to investigate the methodologies and approaches used in past programme impact evaluations as well as to establish the utility value of the evaluation results in offering new insights of what works, for whom,under what conditions and in what respects. Finally, in-depth interviews with policy decision-makers,commissioners and implementers of evaluations were completed to further ascertain the utility value of evaluation results as well as establish the applicability of the REM as a methodological approach in conducting programme impact evaluations in the South African public sector. The overall findings provided evidence that initial well-defined and coherent programme theory as well as programme causality on commissioned impact evaluations are largely absent. The researchalso found limited contextual understanding of the programmes' intersection with the broadercomplex social system. This offered policy-makers limited insights in terms of understanding for whom a social programme will work most effectively or not and the reasons thereof. Therefore, this limited explanatory focus resulted in impact evaluations that had a 'black box' phenomenon, as the key change mechanism in programmes were unknown. It was concluded that REM has a potential value to contribute to programme impact evaluations that offer new insights regarding what works, for whom, under what conditions and in what respects. However, there are potential constraints in its application and these should be well consideredagainst the benefits that can be derived from such evaluations. A key contribution of the study is an assessment model, applicable to the South African context, that could be applied to determine, from a Realist Evaluation lens, the value of an evaluation to different policy-makers' needs and can be applied to assess the limitations of other impact evaluationmethods.

[发布机构] Stellenbosch University