Information Theory
Catalog Data: 

Graduate Course Information


ECE 636

Information Theory



Course Website:

UA Catalog Description:

Course Assessment:

Homework:  8 – 10 assignments.

Exams:  2 Midterm Exams, 1 Comprehensive Final Exam

Grading Policy:

Typically: 10% Homework,

                  30% Exam #1,

                  30% Exam #2,

                  30% Final Exam.

Course Summary:

Definition of a measure of information and study of its properties; introduction to entropy, mutual information, channel capacity, and rate-distortion theory.

ECE503 - Probability and Random Processes

“Elements of Information Theory,” by T.M. Cover and J.A. Thomas, Wiley.

Course Topics: 

1.     Entropy, Relative Entropy, Mutual Information

a. Intuitive interpretation of entropy

b. Joint Entropy

c. Conditional Entropy

d. Conditional Relative Entropy

e. Conditional Mutual Information

f. Sufficient statistics

g. Fano’s Inequality

2.     Asymptotic Equipartion Principle

a. Application of AEP: Simple minded data compression

3.     Entropy Rates of Random Processes

a. Hidden Markov Models

4.     Data Compression

a. Huffman codes

b. Optimality of Huffman codes

c. Shannon-Fano-Elias coding

d. Arithmetic coding

5.     Channel Capacity

a. Binary erasure channel (BEC)

b. Channel Coding Theorem

6.     Differential Entropy

7.     The Gaussian Channel

a. Binary amplitude modulation

b. Independent channels with average power constraint

c. Channels with dependent noise

8.     Rate Distortion Theory

Class/Laboratory Schedule: 

Lecture:  150 minutes/week

Prepared by: 
Michael Marcellin
Prepared Date: 
April 2013

University of Arizona College of Engineering