A comparison of similarity metrics for e-assessment of MS Office assignments

[摘要] English: Computerised assessment is prevalent in various disciplines where immediate and accurate feedback with regard to students' assignments is required. It is used as an alternative to manual assessment of computer programming assignments, computer proficiency tests and free-text responses to questions. The implementation of the Office Open XML (OOXML) standard, as the default document format for Microsoft Office, instigated the development of alternative computerised assessment algorithms with the ability to assess word-processing documents of the DOCX format. Word-processing assignments are primarily assessed by comparing the final document, submitted by the student, to the ideal solution provided by the examiner. Research into the anatomy of OOXML-based documents delivered several alternative approaches with regard to the computerised assessment of DOCX document types. OOXML simplifies the evaluation process of word-processing documents by providing easily identifiable elements within the document structure. These elements can then be used to assess the content and formatting of the document to determine whether the solution, submitted by the student, matches the ideal solution provided by the examiner. By examining current OOXML-based algorithms, certain gaps within the implementation thereof were identified. An alternative algorithm, dubbed the OOXML algorithm that could alleviate these issues, is introduced. It improves the assessment techniques of current OOXML-based algorithms by firstly simplifying the structure of the DOCX documents to ensure that the student's document and examiner's solution conform to a homogeneous structure. It then identifies corresponding paragraphs between the student's document and the examiner's solution. Finally, the student's simplified document is assessed by comparing the content and formatting elements within the OOXML structure of the corresponding paragraphs with one another. To determine the accuracy and reliability of the proposed OOXML algorithm, it is compared with three established algorithms as well as manual assessment techniques. The three algorithms include a string comparison algorithm called fComp, the Levenshtein algorithm and a document difference algorithm, implemented by a system called Word Grader. The same group of word-processing assignments is graded by the specified algorithms and manually assessed by multiple human markers. Analysis of the results of a quasi-experimental study concluded that the proposed OOXML algorithm and its element comparison metric not only produced more reliable results than the human markers but also more accurate results than the human markers and the other selected document analysis algorithms.

[发布机构] University of the Free State