Cooperation will be the motivating factor in a free society.
--Abbie Hoffman

  Information theory

Central | Recent Changes | The Network | Talk unknown user

involves communication of data

the raw stuff of knowledge

the theory developed by Claude Shannon in The Mathematical Theory of Communication

studied by Norbert Wiener

a component of cybernetics

discussed in Human Use of Human Beings

       The fundamental problem of communication is that of
       reproducing at one point either exactly or approximately a
       message selected at another point. Frequently the
       messages have meaning; that is they refer to or are
       correlated according to some system with certain physical
       or conceptual entities. These semantic aspects of
       communication are irrelevant to the engineering problem.
       The significant aspect is that the actual message is one
       selected from a set of possible messages. The system
       must be designed to operate for each possible selection,
       not just the one which will actually be chosen since this is
       unknown at the time of design. 

       The ratio of the entropy of a source to the maximum value it
       could have while still restricted to the same symbols will
       be called its relative entropy. This, as will appear later, is
       the maximum compression possible when we encode into
       the same alphabet. One minus the relative entropy is the
       redundancy. The redundancy of ordinary English, not
       considering statistical structure over greater distances
       than about eight letters, is roughly 50%. This means that
       when we write English half of what we write is determined
       by the structure of the language and half is chosen freely.
       The figure 50% was found by several independent
       methods which all gave results in this neighborhood. One
       is by calculation of the entropy of the approximations to
       English. A second method is to delete a certain fraction of
       the letters from a sample of Enlgish tect and then let
       someone attempt to restore them. If thay can be restored
       when 50% are deleted the redundancy must be greater
       than 50%. A third method depends on certain known
       results in cryptography. 

       Two extremes of redundancy in English prose are
       represented by Basic English and by James Joyce's book
       Finnegan's Wake. The Basic English vocabulary is limited
       to 850 words and the redundancy is very high. This is
       reflected in the expansion that occurs when a passage is
       translated into Basic English. Joyce on the other hand
       enlarges the vocabulary and is alleged to achieve a
       compression of semantic content. 

       The redundancy of language is related to the existence of
       crossword puzzles. If the redundancy is zero any sequence
       of letters is a reasonable text in the language and any
       two-dimensional array of letters forms a crossword puzzle.
       If the redundancy is too high the language imposes too
       many constraints for large crossword puzzles to be
       possible. A more detailed analysis shows that if we
       assume the constraints imposed by the language are of a
       rather chaotic and random nature, large crossword puzzles
       are just possible when the redundancy is 50%. If the
       redundancy is 33%, three-dimensional corssword puzzles
       should be possible, etc. 

       --The Mathematical Theory of Communication
Central | Recent Changes | The Network | Talk unknown user
unknown userRSS