Study Notes on Lexical Analysis

By BYJU'S Exam Prep

Updated on: September 25th, 2023

Lexical analyzer reads the source program character by character and returns the tokens of the source program. It puts information about identifiers into the symbol table.

The Role of Lexical Analyzer:

  • It is the first phase of a compiler
  • It reads the input character and produces an output sequence of tokens that the Parser uses for syntax analysis.
  • It can either work as a separate module or as a submodule.
  • Lexical Analyzer is also responsible for eliminating comments and white spaces from the source program.
  • It also generates lexical errors.


  • Lexical Analyzer is also responsible for eliminating comments and white spaces from the source program.
  • It also generates lexical errors.

Tokens, Lexemes and Patterns

  • A token describes a pattern of characters having same meaning in the source program such as identifiers, operators, keywords, numbers, delimiters and so on. A token may have a single attribute which holds the required information for that token. For identifiers, this attribute is a pointer to the symbol table and the symbol table holds the actual attributes for that token.
  • Token type and its attribute uniquely identifies a lexeme.
  • Regular expressions are widely used to specify pattern.

Tokens, Patterns and Lexemes

Lexeme: A lexeme is a sequence of alphanumeric characters that is matched against the pattern for a token. 
Pattern: The rule associated with each set of string is called pattern. Lexeme is matched against pattern to generate token.
Token: Token is word, which describes the lexeme in source pgm. Its is generated when lexeme is matched against pattern. Example: Lexeme: A1, Sum, Total
  • Pattern: Starting with a letter and followed by letter or digit but not a keyword.
  • Token: ID

Lexeme: If | Then | Else

  • Pattern: If | Then | Else
  • Token: IF | THEN | ELSE

Lexeme: 123.45

  • Pattern: Starting with digit followed by a digit or optional fraction and or optional exponent
  • Token: NUM

Counting Number of tokens :

A token is usually described by an integer representing the kind of token, possibly together with an attribute, representing the value of the token. For example, in most programming languages we have the following kinds of tokens.

  • Identifiers (x, y, average, etc.)
  • Reserved or keywords (if, else, while, etc.)
  • Integer constants (42, 0xFF, 0177 etc.)
  • Floating point constants (5.6, 3.6e8, etc.)
  • String constants (hello there\n, etc.)
  • Character constants (‘a’, ‘b’, etc.)
  • Special symbols (( ) : := + – etc.)
  • Comments (To be ignored.)
  • Compiler directives (Directives to include files, define macros, etc.)
  • Line information (We might need to detect newline characters as tokens, if they are syntactically important. We must also increment the line count, so that we can indicate the line number for error messages.)
  • White space (Blanks and tabs that are used to separate tokens, but otherwise are not important).
  • End of file

Each reserved word or special symbol is considered to be a different kind of token, as far as the parser is concerned. They are distinguished by a different integer to represent their kind.

Example :



So that was all about lexical analysis. Now practice the questions from the app.

You can follow the detailed champion study plan for GATE CS 2022 from the following link:

Detailed GATE CSE 2022 Champion Study Plan

Candidates can also practice 110+ Mock tests for exams like GATE, NIELIT with BYJU’S Exam Prep Test Series check the following link:

Click Here to Avail GATE CSE Test Series!(100+ Mock Tests)

Get unlimited access to 21+ structured Live Courses all 112+ mock tests with Online Classroom Program for GATE CS & PSU Exams:

Click here to avail Online Classroom Program for Computer Science Engineering


Download BYJU’S Exam Prep, Best gate exam app for Preparation
Our Apps Playstore
SSC and Bank
Other Exams
GradeStack Learning Pvt. Ltd.Windsor IT Park, Tower - A, 2nd Floor, Sector 125, Noida, Uttar Pradesh 201303
Home Practice Test Series Premium