Toward a theory of information processing

[摘要] Information processing is performed when a system preserves some aspects of the information encoded by its input while it suppresses others. To describe a system;;s information processing capability, input and output need to be compared in a way invariant to the signal;;s form and how it represents information. We describe an approach to quantify information processing based on applying controlled changes to the input and observing the corresponding output. Information-theoretic distance measures---those that reflect the data processing theorem---are calculated on the input and output separately and compared. The best candidate for the distance measure to describe signal processing is the information theoretic distance measure called the Kullback-Leibler distance (also known as the relative entropy). Properties of the resulting information transfer ratio are used to derive fundamental information processing properties of systems and interconnected systems.

[发布机构] Rice University

[效力级别] Electrical engineering