Standard setting for specialist physician examinations in South Africa

[摘要] English: Setting defensible and fair pass standards for high-stakes postgraduate specialistcertification examinations is a critical quality assurance component of assessment.Doing so in a feasible and sustainable way, within a resource-constrained contextsuch as South Africa, is challenging.Traditionally the 28 member Colleges of the Colleges of Medicine of South Africa(CMSA), the national specialist licensing examination body in South Africa, haveused a fixed pass mark of 50%. This practice does not acknowledge the inherentvariance in examination difficulty and so increases the risk of failing competentcandidates (false negative outcome) and passing incompetent examinees (falsepositive outcome). In 2011, the College of Physicians (CoP), a large CMSA memberCollege, addressed the matter by implementing a standard setting process for thewritten components of their specialist physician certification examinations.The aim of this study was twofold: i) To evaluate the knowledge, attitudes, views andperspectives of CoP examiners regarding standard setting, and ii) compare theperformance and utility of the Cohen and Angoff methods to advise the CoP regardingan appropriate standard setting method in a resource-constrained setting.A literature review was done to conceptualise standard setting as it pertains toassessment in medical education. In addition, policies and regulatory systems relevantto specialist certification examinations in South Africa were reviewed to provide thecontext for this study.Two research components were concurrently conducted between 2012 - 2014:A prospective study evaluated the knowledge, attitudes, views and perspectives ofCoP examiners regarding standard setting before and after training and 30 months of practical experience using both the Cohen and Angoff methods of standardsetting.A comparative study evaluated the performance (pass marks and failure rates) andutility (according to a framework derived from the literature review) of the Cohenand Angoff methods using five cycles of examination data, including multiplechoice questions (MCQ), short answer questions and short essay questions.The introduction of standard setting was successful and widely supported by the CoPexaminers. The Cohen method performed well when used for test data with areasonable number of test items (30 or more) in homogeneous exit-level cohorts ofmore than 50 candidates. Tests containing few test items (i.e. short essay questions)performed poorly. The performance of the Cohen method was variable for smallercohorts (less than 100) of candidates drawn from heterogeneous populations, such asentry-level Part I MCQ test takers. The Angoff method yielded unacceptable outcomesregardless of test format. The utility comparison identified the Cohen method as thepreferred standard setting method for the CoP.The findings of this study support the introduction and ongoing use of the Cohenmethod as a feasible and sustainable method of setting pass marks for the writtencomponents of the CoP certification examinations. Education and training in the use ofstandard setting methods, as part of a change management strategy, improvedexaminers' understanding of the role, importance and basic methodology of standardsetting and strengthened their support for the use of standard setting in certificationexaminations. More data are needed to evaluate the true impact of cohort size on thestability of the Cohen method for entry-level, heterogeneous cohorts of examinees.The purist Angoff strategy, used in this study due to resource limitations, performedpoorly and was deemed 'not fit for purpose' by the CoP examiners. The usefulness ofthe novel standard setting utility framework developed in this study warrants furtherresearch in other examination settings such as performance–based examinations.

[发布机构] University of the Free State