Information theory

  • Entropy and self-information
  • Conditional entropy
  • Joint entropy and mutual information
  • Parallels to probability theory
  • Physical interpretation of entropy

KL Divergence