A comparative study to determine the optimum e-assessment paradigm for testing users' word processing skills

[摘要] English: In recent times, people have become more and more reliant on computers on a daily basis. As a result, the need has arisen to optimise the task-related experience in terms of time efficiency, which demands effective training in software skills. To be more specific, word processing skills are currently considered essential in any field of work and are in high demand. This study focuses on determining the optimal paradigm (methods) to assess users' word processing skills. One of the main reasons for this research was the fact that students at the University of the Free State (UFS) reported to the computer literacy course lecturer that they were dissatisfied with the virtual, simulated MS Word software environment used to assess (e-assess) their word processing skills electronically. This existing test system (ETS) at the UFS requires students to perform certain tasks and automatically checks whether the required end-result is obtained. However, this system is based on a simulated interface with limited functionality. As a result, the relevant information on software e-assessment systems was researched and a new software skills e-assessment application developed accordingly. The aim was to develop a tool that would be able to assess students' word processing skills in the most reliable way possible. Another aim was to find methods of stimulating the learning process during the e-assessment of word processing skills. Therefore the newly developed e-assessment system, WordAssessor, is designed to be based on the real MS Word environment. It requires students to perform certain tasks and automatically checks whether the said tasks have been correctly performed. WordAssessor allows students to explore the MS Word interface fully while being assessed. It even allows students to use trial-and-error to solve certain problems (tasks). To stimulate learning potentially further during e-assessment, WordAssessor presents students with a video solution for the questions they had incorrect, directly after the test. In order to assess the validity of the e-assessment methods employed by the WordAssessor system, comparative user testing was conducted. Students' word processing skills were assessed as part of their advanced computer literacy course, using the ETS, the WordAssessor system and a personalised test scenario (where no e-assessment tool was used). In addition, participating students were provided with a questionnaire to determine their reaction and preference with regard to the various elements of the assessment methods. By analysing the results in detail, it was found that the results of the personalised test scenario (PT) yielded the most reliable indication of students' true word processing skills, and could be used as a benchmark. Hereafter, the results of the WordAssessor test scenario were analysed to determine the correlation (relationship) with the results of the personalised test. The same type of correlation was performed between the results of the ETS and PT. It was established that the WordAssessor results correlated significantly more closely with the PT results than the ETS. In the end (and after additional analysis) it was found that the methods employed by WordAssessor yielded the most reliable indication of students' true word processing skills knowledge when compared to the ETS. In addition, from the results of the post-assessment questionnaire, it was determined that students felt they learned more as a result of the video tutorials. Furthermore, they stated that they preferred video feedback over text- or paper-based feedback. They also stated that they preferred being assessed in a real software environment, as opposed to a simulation. It was recommended that a more flexible and realistic e-assessment approach (as demonstrated by the WordAssessor system) could be beneficial to students on several levels. Also, it was recommended that students be able to answer test questions in a way that suits them rather than being instructed as to the method of use. Finally, the use of highly detailed video tutorials directly following e-assessment (only for questions students had incorrect), was recommended.

[发布机构] University of the Free State