Information Theory

By : Vandana Tanwar

Updated : Feb 4, 2021, 13:26

Information theory is one of the essential topics to cover in digital communications for the GATE exam, along with multiplexing and decoding techniques. Information is necessary for communication, be it analog or digital. Information theory refers to a mathematical model to study coding techniques and storage and quantification of information. This article covers essential details of information theory for GATE ECE, a list of best information theory books, Information theory notes for ECE, and information theory GATE questions.

This topic is not only mandatory for GATE exams, but also for other engineering exams.

Important Information Theory Topics for GATE ECE

Mentioned below are a few vital concepts that come under information theory, which will help you prepare for questions on it.




Entropy refers to the measure of the average data content per source symbol. American electrical engineer Claude Shannon, who is knowns as the father of the Information theory, proposed a formula:

S = kB ln W


S= entropy

kB= Boltzmann constant

In= natural logarithm

W= Number of microscopic configurations

Mutual information

Mutual information refers to the number of quantities that measure how much a single random variable can tell us about another. It is considered a dimensionless quantity with the standard units of Bit. It can be understood as depletion in uncertainty about a random variable if another variable’s information is given. High mutual information shows a huge reduction in variability, whereas low mutual information highlights little reduction. If zero mutual information is indicated between 2 random variables, it means both are independent.

Channel capacity

Channel capacity is a commonly used metric used to indicate the maximum quantity of traffic or signal transmitted over a particular infrastructure channel. A given channel’s channel capacity refers to the highest information rate that can be attained with only a little probability of error. It is represented by the symbol ‘C’ and is computed in bits per channel use

Discrete memoryless source

A discrete memoryless source refers to a source from which the information is being discharged at successive intervals, independent of preceding values. It is termed ‘discrete’ as it is not considered for a continuous-time interval. It is considered memoryless because it is new at each instant of time.

Tips to Solve Information Theory Questions for GATE ECE

  • Start with analog communication and learn all the basics of digital communication.
  • Solve information theory online tests and practice related MCQ questions.
  • Information theory overlaps with communication theory, so make sure you cover the latter, too.
  • Take out time for revision.

Importance of Information Theory for GATE 

Here are a few points explaining why it is crucial to cover information theory topics for GATE ECE:

  • Information theory comes under digital communication, which is an essential topic for the GATE exam.
  • The GATE exam carries at least nine marks questions based on information theory.
  • Approximately 9% of the GATE 2020 paper had questions on communication systems. 
  •  It is crucial to cover information theory to understand analog communication. 

Most Recommended Books for Information Theory for GATE

Listed below are some of the best books to cover digital communication and information theory for the GATE 2021 exam:



Digital Communication


Digital Communication

Modern Digital and Analog Communication Systems 

Simon S. Haykin

 B.P. Lathi 

Why Prepare Information Theory from BYJU'S Exam Prep?

Today, BYJU'S Exam Prep is one of the best platforms to prepare for competitive exams as it offers detailed notes covering each topic, PDFs, practice question papers, and online quizzes. You can rely on BYJU'S Exam Prep for GATE exam preparation as the complete information theory GATE syllabus is available on the website along with notes, pdfs, MCQs, and plenty of practice tests. Any aspirant can prepare better with BYJU'S Exam Prep’s comprehensive video lectures and live interactive lessons. Its day-wise preparation plan is precisely prepared to fit a student’s needs.


Q1 - What all topics need to be covered besides information theory for the GATE exam?

GATE topics include information theory, random variable, encoding scheme, Shannon capacity, information rate, and the probability of error.

Q2 - What is an excellent book to learn about signals and systems?

There are many excellent communication theory books; refer to ‘Signals and Systems’ (Edition 2) by Alan V. Oppenheim, Alan S.Willsky, and S. Hamid Nawab to cover this topic. 

Q3- Does the GATE exam comprise general aptitude questions? 

Yes, general aptitude carries approximately 15% weightage in the GATE exam. 

Q4- How to revise and practice information theory topics?

To practice information theory topics, visit BYJU'S Exam Prep, and solve online tests. Appear for information theory online quizzes to prepare well for the GATE exam. 

Q5- How many times is the GATE exam conducted each year?

GATE exam is conducted once every year, usually in February.