A vision-based South African sign language tutor

[摘要] ENGLISH ABSTRACT: A sign language tutoring system capable of generating detailed context-sensitive feedbackto the user is presented in this dissertation. This stands in contrast with existing sign languagetutor systems, which lack the capability of providing such feedback.A domain specific language is used to describe the constraints placed on the user's movementsduring the course of a sign, allowing complex constraints to be built through the combinationof simpler constraints. This same linguistic description is then used to evaluate theuser's movements, and to generate corrective natural language feedback. The feedback is dynamicallytailored to the user's attempt, and automatically targets that correction which wouldrequire the least effort on the part of the user. Furthermore, a procedure is introduced whichallows feedback to take the form of a simple to-do list, despite the potential complexity of thelogical constraints describing the sign. The system is demonstrated using real video sequencesof South African Sign Language signs, exploring the different kinds of advice the system canproduce, as well as the accuracy of the comments produced.To provide input for the tutor system, the user wears a pair of coloured gloves, and a videoof their attempt is recorded. A vision-based hand pose estimation system is proposed whichuses the Earth Mover's Distance to obtain hand pose estimates from images of the user's hands.A two-tier search strategy is employed, first obtaining nearest neighbours using a simple, butrelated, metric. It is demonstrated that the two-tier system's accuracy approaches that of aglobal search using only the Earth Mover's Distance, yet requires only a fraction of the time.The system is shown to outperform a closely related system on a set of 500 real images ofgloved hands.

[发布机构] Stellenbosch University