Jeff Liu Lab
Home
Projects
Workshop
AI Wiki
AI Lab
Shop
中
Sign In
All
Computing Science
Artificial Intelligence
Deep Learning
Reinforcement Learning
AI Agents
Embodied Intelligence
Robot Engineering
Human-Like Intelligence
AI Engineering
← Back to Wiki
Computing Science
Mathematical Foundations
Calculus
Linear Algebra
Probability Theory
Information Theory
Statistics
Automatic Differentiation
Discrete Mathematics
Numerical Methods
Optimization Theory
Graph Theory Fundamentals
Theory of Computation
Algorithms
Computer Architecture
Operating Systems
Computer Networks
Programming Languages
Software Engineering
Comments (0)
Sign in to comment
Table of Contents
Entropy
Self-Information
Cross-Entropy
KL Divergence (Relative Entropy)
Conditional Entropy
Joint Entropy
Mutual Information
Relationship to Other Entropy Measures
Relationships Among Entropy Measures (Venn Diagram)
Information Gain
Information Bottleneck
Core Idea
IB-Lagrangian
Connection to Deep Learning
Maximum Entropy Principle
Intuition
Mathematical Formulation
Common Examples
Applications in Machine Learning
Comments
Comments (0)
Sign in to comment