Shannon, 1948: measuring information to communicate reliably

Written by AI from the cited sources. Editorial method and checks.

AI-generated conceptual illustration, not an archival photograph or a quantitative diagram. Relay-like devices and light pulses evoke communication through noise; they do not document a particular 1948 apparatus.

1. A common problem behind different machines

A telegram, a spoken sentence and a photograph look different, but communication gives them a shared engineering problem: reproduce a selected message somewhere else. A cable, radio wave or storage device is a physical means of doing this. The receiver must distinguish alternatives despite imperfections in the medium.

Shannon's paper appeared in two parts in the Bell System Technical Journal in July and October 1948. It built on earlier work, including Nyquist and Hartley, while treating both source statistics and noise. The MIT copy cited here is explicitly a corrected reprint, not a facsimile of the two original issues. S1, S2

Information in this theory concerns uncertainty about which message was selected. It is not a score of meaning, truth, usefulness or intelligence. A predictable warning may matter enormously while containing little new statistical information. This restriction makes the engineering question tractable; it does not make meaning unimportant.

2. From uncertainty to bits

Let X be a symbol drawn from a finite alphabet, with probabilities p_i that sum to one. The surprise of outcome i and its average, the entropy, are:

$I_i=-\log_2 p_i.$Ii​=−log2​pi​.

$H(X)=-\sum_i p_i\log_2 p_i.$H(X)=−i∑​pi​log2​pi​.

All logarithms here use base 2, so entropy is measured in bits per source symbol; the convention is 0 log₂0 = 0. An equiprobable choice between two outcomes has entropy 1 bit. A certain outcome has entropy zero. A bit is a unit of information, not a particular voltage or a synonym for one printed character.

Consider an illustrative memoryless source with five symbols. Assign shorter codewords to more likely symbols:

$\begin{aligned} H(X)=\bar L&=\tfrac12(1)+\tfrac14(2)\\ &\quad+\tfrac18(3)+2\tfrac1{16}(4)\\ &=1.875. \end{aligned}$H(X)=Lˉ​=21​(1)+41​(2)+81​(3)+2161​(4)=1.875.​

No codeword starts another codeword, so this prefix code can be decoded without separators. For 1,000 independent symbols, the expected length is 1,875 bits, compared with 3,000 bits for a fixed three-bit label per symbol: a 37.5% reduction. These are expected lengths under the stated distribution, excluding headers and transmission overhead—not a guarantee for every message. A rare symbol actually takes four bits. S3

3. Compression has a precise limit

For a known, finite, independent and identically distributed source, the best binary prefix code for blocks of n symbols has expected block length L_n satisfying:

$nH(X)\le L_n<nH(X)+1.$nH(X)≤Ln​<nH(X)+1.

$H(X)\le \frac{L_n}{n}<H(X)+\frac1n.$H(X)≤nLn​​<H(X)+n1​.

Dividing by n shows how the overhead above H(X) can become small as blocks grow. The lower bound also applies to uniquely decodable codes. This is lossless coding: the original message is reconstructed exactly. With correlations, the relevant quantity is an entropy rate under appropriate source assumptions, not automatically the entropy of one isolated symbol. S3

Compression therefore exploits predictable structure; it cannot make every possible binary file shorter. There are fewer short strings than long ones. Some inputs must stay the same size or grow. A codebook learned from the wrong distribution, a tiny file, or the cost of transmitting that codebook can erase the practical saving.

4. Adding redundancy to withstand noise

Source coding removes avoidable redundancy. Channel coding deliberately adds structured redundancy so the receiver can distinguish messages after perturbations. There is no contradiction: the two operations solve different problems. The schematic separates them for explanation; physical modulation and synchronization are omitted.

Scientific schematic created for this article, inspired by the communication-system abstraction in Shannon: compression precedes protective coding; decoding reverses their order. Arrows show the message flow, not a measured signal or a time scale.

For a simple calculation, suppose each transmitted bit flips independently with probability p = 0.1. Send a logical 0 as 000 and a logical 1 as 111, then decode by majority. An error needs at least two flips:

$P_{\rm e}=3p^2(1-p)+p^3=0.028.$Pe​=3p2(1−p)+p3=0.028.

The decoded bit error probability falls from 10% to 2.8%, at a rate of one information bit per three channel uses. This repetition code demonstrates the trade-off but does not approach Shannon's limit. Its residual error does not vanish merely by sending more independent three-bit blocks. Bursty or correlated errors would invalidate this particular calculation.

5. Capacity: a rate, not a promise of perfection

For the finite-alphabet, stationary discrete memoryless channel used here, the conditional law of each output Y depends only on its corresponding input X. With a specified transition law and unconstrained input distribution, capacity is:

$I(X;Y)=H(X)-H(X\mid Y),$I(X;Y)=H(X)−H(X∣Y),

$C=\max_{P_X} I(X;Y).$C=PX​max​I(X;Y).

$R=\frac{\log_2 M}{n}.$R=nlog2​M​.

In these formulas, M is the number of equiprobable messages and n the number of channel uses in one codeword. In the source-coding bound, n instead counts source symbols and L_n is an expected length in bits. These two uses of n refer to separate examples.

Mutual information I(X;Y) measures how much observing Y reduces uncertainty about X. Capacity is in bits per channel use. For equiprobable messages encoded into length-n blocks, every fixed rate R strictly below C admits sequences of codes whose block error probability tends to zero as n grows. Above C, vanishing error is impossible in this model. This is an asymptotic existence result, not a guarantee of zero errors at finite length, or of an efficient implementation. S4, S5, S6

For the binary symmetric channel with independent flip probability p:

$\begin{aligned} h_2(p)&=-p\log_2 p\\ &\quad-(1-p)\log_2(1-p). \end{aligned}$h2​(p)​=−plog2​p−(1−p)log2​(1−p).​

$C=1-h_2(p).$C=1−h2​(p).

$C(0.1)\approx0.5310.$C(0.1)≈0.5310.

Equiprobable input bits attain this capacity. At p = 0.1, the limit is about 0.5310 information bits per use—not 0.9. Unknown error locations consume distinguishability. At 1,000 uses per second, 531.0 bit/s is the asymptotic capacity, not an application throughput measurement.

6. When bandwidth and power enter the calculation

For an ideal band-limited additive white Gaussian noise channel, with noise independent of the signal and a constraint on average signal power, the familiar result is:

$C=W\log_2\left(1+\frac{P}{N}\right).$C=Wlog2​(1+NP​).

$\begin{aligned} 3000\log_2(1001)&\\ &\approx29901.7\;\mathrm{bit/s}. \end{aligned}$3000log2​(1001)​≈29901.7bit/s.​

Here W is bandwidth in hertz; P and N are signal power and total noise power within that bandwidth, in the same unit such as watts. P/N is a dimensionless linear power ratio, not a decibel value. With W = 3,000 Hz and a signal-to-noise ratio of 30 dB, P/N = 10^(30/10) = 1,000, giving approximately 29,901.7 bit/s, or 29.90 kbit/s. These are illustrative model inputs, not a measured historical telephone-link performance. S1, theorem 17

The formula is not universal for fading, impulsive interference, amplitude limits or a prescribed finite constellation. Increasing W can also increase N: the noise power must be recomputed rather than held fixed without justification. Hardware, coding length, delay and protocol overhead determine how far an implementation lies below the ideal limit.

7. What changed—and what the theory does not say

Shannon made it possible to ask two separate quantitative questions: how much information a source generates, and how much a channel can reliably carry. A design can then be compared with a limit rather than judged only against an earlier machine. The source and channel models must still be defended with measurements.

The 1948 work did not supply every practical compression algorithm or error-correcting code. Later work, including Gallager's analysis of coding error probabilities, developed the connection between rate, block length and reliability. Nor does the theory say that random data are meaningful, that compression proves understanding, or that greater computing power can defeat a channel's capacity without changing its assumptions. S6

Its enduring achievement is a language for reasoning about uncertainty under constraints. The useful engineering response to a limit is to identify the model, measure the gap, and decide whether to improve the code, the channel, or the objective.

Sources and checks for this article

English article written by an AI system from the cited sources. The same system produced the seven translations and scientific schematic. Numerical examples were recalculated by code and the versions were checked by AI; no independent expert or human review is claimed. The cover was generated with ChatGPT Image.

Reference revision SHANNON-EN-1, 9 September 2026. The cited 1948 passages and later mathematical treatments were read online. Equations, assumptions, arithmetic, codeword uniqueness and localized labels were checked. Examples are theoretical calculations, not experiments. The explanatory schematic is distinct from the generated cover. This note describes this article’s actual preparation; it does not certify autonomous publication.

Sources

• S1 — C. E. Shannon — A Mathematical Theory of Communication (1948), corrected reprint
• S2 — Nokia Bell Labs — A Mathematical Theory of Communication
• S3 — M. Médard, MIT 6.441 (2010) — Lecture 5: codes, Kraft inequality, optimal codes
• S4 — M. Médard, MIT 6.441 (2010) — Lecture 8: channel capacity
• S5 — M. Médard, MIT 6.441 (2010) — Lecture 10: channel coding theorem
• S6 — R. G. Gallager — A Simple Derivation of the Coding Theorem and Some Applications (1965)