Research and Implementation of Chinese Text Automatic Proofreading System

[摘要] The news media platform has a huge amount of original news releases every day, it is impractical to use manual review of text typos. This paper designed and implemented a Chinese text automatic proofreading system for large-scale text content and high-speed processing. The proofreading content is first analyzed and classified: typos and sensitive information. Firstly, the system used the n-gram model to statistically analyze the corpus after segmentation to form a 2-gram model library and a contextual context library; secondly, builded a typo confusion set, and then calculated the probability of the target word in the knowledge base to realize automatic error detection and correction of Chinese text. The system has been successfully applied to the error of the content of many government news media platforms, each server can handle one million articles every day. The results show that the recall rate of the article is 78.9% and the accuracy rate is 85.1%. It meets the demand of high speed and accurate processing of massive text error, and has important practical significance and application fields.

[发布机构] Beijing Key Laboratory of Big Data Technology for Food Safety, School of Computer and Information Engineering, Beijing Technology and Business University, Beijing; 100048, China^1

[效力级别] 无线电电子学 [学科分类] 计算机科学（综合）

[关键词] Accuracy rate;Application fields;Error detection and correction;High-speed processing;Knowledge base;Model library;N-gram modeling;Sensitive informations