optional · CSS study notes
Lexical Analysis — Input Buffering & Token Specification
Definition
The lexical analyzer (scanner) reads raw source characters and groups them into tokens — the smallest meaningful units (keywords, identifiers, operators, literals).
Key Points
- Input Buffering: source is read in large blocks (not char-by-char) to reduce I/O overhead. Double-buffering scheme uses two buffers with a sentinel character (EOF marker) to detect buffer boundaries efficiently without extra checks per character.
- Token, Lexeme, Pattern:
- Token: category name, e.g.
ID,NUM - Lexeme: actual character sequence, e.g.
total,42 - Pattern: rule describing the lexeme, usually a regular expression
- Token: category name, e.g.
- Specification of tokens: done using regular expressions/regular definitions.
- Recognition of tokens: implemented via finite automata (transition diagrams) built from the regex patterns.
- Handles removal of whitespace, comments; tracks line numbers for error reporting.
Example
For total = rate + 12:
| Lexeme | Token |
|---|---|
| total | ID |
| = | ASSIGN_OP |
| rate | ID |
| + | ADD_OP |
| 12 | NUM |
📌 CSS Frequency: High (explicitly listed as 2016 repeated topic) — often paired with "define token, lexeme, pattern" and buffering diagrams.
Model Answer (short)
"Lexical analysis converts a character stream into tokens using patterns specified as regular expressions and recognized via finite automata. Input buffering (commonly double-buffering with sentinels) minimizes the overhead of re-reading characters. A token is the category (e.g. ID), a lexeme is the actual matched text (e.g. 'total'), and a pattern is the rule (regex) that defines the token."
📜 Past-Paper Questions & MCQs (CSS 2016–2026)
2020 · Paper-I · Q7(a) · 5 marks
Q. How many tokens are there in printf("k = %d, &k = %x", k, &k);?
Ans.
- Tokens in order:
printf(1),((2), the string literal"k = %d, &k = %x"(3),,(4),k(5),,(6),&(7),k(8),)(9),;(10). - Total: 10 tokens. The whole string in quotes is one token, so the
&,%and,inside it are not counted.
2026 · Paper-I · Q7(b) · 6 marks
Q. Explain the difference among lexeme, token and pattern with examples.
Ans.
| Term | Meaning | Example |
|---|---|---|
| Lexeme | The actual character sequence in the source code | count, 25, + |
| Token | A category (name plus optional attribute) given to a lexeme | ID, NUMBER, PLUS |
| Pattern | The rule describing all lexemes of a token (a regular expression) | ID: letter(letter |
- Example: in
x = 25 + y, the lexeme25belongs to token NUMBER, whose pattern isdigit+.
2026 · Paper-I · Q8(a) · 6 marks
Q. A lexical analyser scans 9var = x + 12abc;. Describe step by step how it processes this input, identifies errors and generates tokens. Which rules of lexical analysis does it use?
Ans.
- Rules used:
- identifier pattern
letter (letter | digit)*(it cannot start with a digit) - number pattern
digit+ - longest match (maximal munch): take the longest string that fits a pattern
- whitespace is skipped, and a lexical error is reported when no pattern matches
- identifier pattern
- Step by step:
9var: starts with a digit, so the scanner reads the number9, but letters follow directly. This is a malformed token (an identifier cannot begin with a digit), so a lexical error is reported.- space skipped
=→ token ASSIGNx→ token ID (entered in the symbol table)+→ token PLUS12abc: a number followed by letters, so a second lexical error.;→ token SEMICOLON
- Tokens produced: ERROR(9var), ASSIGN, ID(x), PLUS, ERROR(12abc), SEMICOLON.
- Recovery: panic mode. Characters are discarded until a delimiter, and scanning continues.
MCQs
2026 P-I · Q18. What do compilers use for recognising tokens?
- (A) Regular expressions ✅
- (B) Context-free grammars
- (C) Machine code
- (D) None of these
- Answer: (A) Regular expressions.
- Why: token patterns (identifiers, numbers) are regular languages, so they are written as regular expressions and recognised by finite automata. Context-free grammars are used later, in the parser, for nested structure.