Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence
How much of a rich source can we discard without changing what its best compressed representation can achieve? A sufficient summary preserves the tradeoff exactly.
Some source detail costs information without helping the task
The Information Bottleneck asks how much information a representation must retain about a source to preserve useful information about a target. A rich source may contain many distinctions that have no bearing on that target.
I ask when those distinctions can be removed before solving the bottleneck problem. If a deterministic summary preserves the full conditional distribution of the target, I prove that working from the summary preserves the entire optimal relevance–rate tradeoff.
Watch the irrelevant information disappear
Take a fair signal bit Z and d independent nuisance bits W. The target C is a noisy copy of Z, flipped with probability 10%. The representation X is another noisy copy of Z. Copying W alongside X increases the source information retained, while adding nothing about C.
Same relevance. Less source information.
Encode the full source
Source (Z, W). Representation (X, W) copies the nuisance.
Encode the sufficient summary
Source Z. Representation X keeps the same target information.
- Rate removed
- 4.000bits
- Source states
- 32 → 2
- Relevance retained
- 0.320bits in both cases
With four nuisance bits and 10% encoder noise, source information falls from 4.531 to 0.531 bits. Target information stays at 0.320 bits.
Calculated from Section 8 of the public preprint, using base-two logarithms. Source bars share a 0–9 bit scale; relevance bars share a 0–1 bit scale. Each symbol represents a source coordinate, not a measured data packet.
The calculation behind the visual
Write h₂ for binary entropy in bits, δ for encoder noise and ε = 0.1 for target noise. The reduced source rate is 1 − h₂(δ). The full representation that copies the nuisance has rate 1 − h₂(δ) + d. Both have target relevance 1 − h₂(ε + δ − 2εδ).
The theorem’s averaging operation replaces the copied nuisance coordinate with independent noise. That independent coordinate can then be omitted without changing either information value. The visual compares the original representation with this simpler equivalent reduced representation.
The result extends beyond this binary example
Let T be the full source and Z = φ(T) a deterministic summary sufficient for the target C. Sufficiency means that knowing T adds no information about C once Z is known.
For any full-source encoder p(X|T), average its behaviour over source values with the same summary. The resulting encoder q(X|Z) preserves the joint distribution of X and C. Its rate saving is exact:
Ip(X; T) − Iq(X; Z) = Ip(X; T | Z)This gives equal optimal IB curves and equal Lagrangian infima at every tradeoff parameter. Where an optimum is attained, the minimisers also correspond. For a finite target with logarithmic loss and independent, identically distributed source blocks, the best remote distortion is preserved at every blocklength and message budget.
What the reduction requires
The summary must be exactly sufficient for the named target. A representation that works for one task can discard information needed by another. Approximate sufficiency does not establish the exact equalities shown here.
The theorem characterises what an appropriate summary preserves. It does not supply a universal procedure for discovering or certifying that summary from limited data. The animation illustrates one finite construction; it is not a benchmark of learned encoders or a claim that arbitrary dimensionality reduction is lossless.
From relevance to coordination
This is part of my research on information requirements for coordination. The allocation and verification paper studies a related practical tension: a finer description can improve allocations while leaving too little evidence in each category. The papers address distinct questions about what information is useful and what observations can support.
Paper and citation
J. Armstrong, “Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence,” arXiv:2604.26744, 2026. This explanation follows v2, revised 2026-08-03. Current public record.