An Introduction to Information Theory

An Introduction to Information Theory. Instructor: Prof. Adrish Banerjee, Department of Electrical Engineering, IIT Kanpur. Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practical source compression algorithms. We will also study how to compute channel capacity of simple channels. (from

Lecture 15 - Channel Capacity

In this lecture, we first explain what is a discrete memoryless channel (DMC). We describe two special cases of DMC, namely, uniformly dispersive channel and uniformly focusing channel and give expressions for their channel capacity. We then define a strongly symmetric channel and give expression for its channel capacity. Finally, we show that a special class of DMC called symmetric channel can be decomposed into strongly symmetric channel and hence their channel capacity can be calculated in terms of capacity of decomposed strongly symmetric channel.

Go to the Course Home or watch other lectures:

Lecture 01 - Introduction
Lecture 02 - Measure Information
Lecture 03 - Information Inequalities
Lecture 04 - Problem Solving Session I
Lecture 05 - Block to Variable Length Coding: Prefix Free Code
Lecture 06 - Block to Variable Length Coding: Bounds on Optimal Code Length
Lecture 07 - Block to Variable Length Coding: Huffman Coding
Lecture 08 - Variable to Block Length Coding
Lecture 09 - The Asymptotic Equipartition Property
Lecture 10 - Block to Block Coding of DMS (Discrete Memoryless Source)
Lecture 11 - Problem Solving Session II
Lecture 12 - Universal Source Coding: Lempel-Ziv Algorithm - LZ77
Lecture 13 - Universal Source Coding: Lempel-Ziv Welch Algorithm (LZW)
Lecture 14 - Coding of Sources with Memory
Lecture 15 - Channel Capacity
Lecture 16 - Jointly Typical Sequences
Lecture 17 - Noisy Channel Coding Theorem
Lecture 18 - Differential Entropy
Lecture 19 - Gaussian Channel
Lecture 20 - Parallel Gaussian Channel
Lecture 21 - Problem Solving Session III
Lecture 22 - Rate Distortion Theory
Lecture 23 - Blahut-Arimoto Algorithm
Lecture 24 - Problem Solving Session IV