Certainty driven consistency loss on multi-teacher networks for semi-supervised learning

[摘要] One of the successful approaches in semi-supervised learning is based on the consistency regularization. Typically, a student model is trained to be consistent with teacher prediction for the inputs under dif-ferent perturbations. To be successful, the prediction targets given by teacher should have good quality, otherwise the student can be misled by teacher. Unfortunately, existing methods do not assess the quality of the teacher targets. In this paper, we propose a novel Certainty-driven Consistency Loss (CCL) that ex-ploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on them in the consistency regularization. We further introduce a novel decoupled framework to encourage model difference. Experimental results on SVHN, CIFAR-10, and CIFAR-10 0 demonstrate the advantages of our method over a few existing methods. (c) 2021 Elsevier Ltd. All rights reserved.

[发布日期] 2021-12-01

[关键词] Semi-supervised learning;Certainty-driven consistency loss;Uncertainty estimation;Decoupled student-teacher;Reliable targets;Noisy labels