Shannon entropy
A measure of the average uncertainty or information content of a random source, in bits.
Also known as: information entropy · Informationsentropie
At a glance
Key signals
20% cross fields · reaches 8 more
- Information Theory
- Explanation
- Examples
- Misconception
- Sourced relations
- Attribution
Dependencies
What this concept builds on and what it makes possible — derived from the atlas’s dependency, causal and structural relations, not from every related edge.
Enables · leads to
Channel capacitydepends onShannon entropyEstablished
The limit of reliable rate.
Mechanism: Shannon's theorem sets channel capacity, the top speed of error-free communication over noise.
Information theorydepends onShannon entropyEstablished
Entropy is its measure.
Mechanism: Information theory measures data in bits of Shannon entropy, bounding compression and communication.
Mutual informationis derived fromShannon entropyEstablished
Shared information.
Mechanism: Mutual information is the drop in one variable's entropy once another is known.
Structural role & consequence
Interpreted from the current atlas graph — what the connections mean, not just how many there are.
20% of its relationships cross field boundaries, reaching 8 other disciplines — unusual in a discipline where most concepts stay within their field.
structural · Structural graph analysis — not a claim of importance, causation or history.
Currently dark in the atlas: 5 of 5 of its relations lack claim-level evidence.
atlas representation · Describes the current Thinking OS representation, not the state of the world.
Structural neighbourhood: 5 → 34 → 126 concepts reachable within 3 hops.
structural · Structural reach — being reachable is not the same as being understood.
cross-field
4 within-field, 1 cross-field
Strengths & constraints
Constraints
- Evidence coverage currently thin in the atlas — few of its relationships carry claim-level evidence. atlas representation
Conditions
- Read structurally — most of its relationships carry no external evidence yet, so claims here are graph-derived. structural
Dependency radial
What this concept builds on (left) and what it makes possible (right) — derived from dependency and causal relations.
What builds on this
2 concepts build on this directly, 3 in total, across 3 disciplines.
Structural downstream reach along dependency edges — not a claim of historical necessity.
Seen through each discipline
How this concept sits in each of its fields — derived from its real connections in the graph, not asserted.
Through this lens it connects to Entropy, Information theory, Channel capacity and Mutual information.
Formula
The average information (in bits) of a discrete random variable.
H(X)— entropy of X (bits)p_i— probability of outcome i
Source: A Mathematical Theory of Communication
Key dates
- 1948PublicationShannon introduces information entropy in “A Mathematical Theory of Communication”. — A Mathematical Theory of Communication
Related ideas to explore
Concepts that look related but are not yet connected here — candidates for a connection to reason about, not established links.
This idea also appears in…
The same structure shows up in other disciplines. These are real recurrences drawn from the graph — a starting point for asking “what carries over, and what changes?”
Concepts
explicitly analogous
crosses a discipline boundary
- Connects 5 other ideas across 1 discipline.
- A cross-disciplinary bridge — its connections reach into 8 other fields.
- Most of its connections are of the “Dependency” kind.
Derived from the graph’s real structure — observations, not a score.
Sources
- The Relationship Between Logical Entropy and Shannon Entropy (2021) verified