The Scanner (also called the lexer) is the first phase of the TypeScript compilation pipeline. It converts raw source text into a stream of tokens, each classified by a SyntaxKind enumeration value. The scanner operates character-by-character through the input text, identifying keywords, identifiers, literals, operators, punctuation, and trivia (whitespace, comments).
For information about how these tokens are assembled into an Abstract Syntax Tree, see Parser.
The scanner's primary responsibility is lexical analysis: transforming a sequence of characters into a sequence of tokens. Each token represents a meaningful unit in the TypeScript language, such as function, 123, "hello", +, or {. The scanner does not understand the grammatical structure of the language—that is the parser's job.
Sources: src/compiler/scanner.ts1-2387 src/compiler/types.ts40-492
The scanner is implemented as a closure returned by createScanner, which maintains internal state as it advances through the source text.
Sources: src/compiler/scanner.ts500-2387 src/compiler/types.ts5566-5596
The Scanner interface defines the public API for lexical analysis. It is defined in types.ts and implemented in scanner.ts.
Key Scanner Methods:
| Method | Purpose |
|---|---|
scan() | Advances scanner position and returns the SyntaxKind of the next token. src/compiler/scanner.ts1200-1800 |
getToken() | Returns the current token's SyntaxKind without advancing. src/compiler/scanner.ts600-610 |
getTokenPos() | Returns the starting position of the current token. src/compiler/scanner.ts612-615 |
getTextPos() | Returns the current scanning position (after the token). src/compiler/scanner.ts617-620 |
getTokenText() | Returns the raw text of the current token. src/compiler/scanner.ts625-630 |
getTokenValue() | Returns the interpreted value (e.g., the content of a string literal). src/compiler/scanner.ts632-635 |
getTokenFlags() | Returns TokenFlags (e.g., Unterminated, PrecedingLineBreak). src/compiler/scanner.ts637-640 |
setText() | Resets the scanner with new source text. src/compiler/scanner.ts550-565 |
Sources: src/compiler/types.ts5566-5596 src/compiler/scanner.ts500-650
The SyntaxKind enum in types.ts defines all possible tokens. The scanner produces token-level kinds, while the parser produces node-level kinds.
Token Ranges:
SingleLineCommentTrivia (43) to ConflictMarkerTrivia (51). src/compiler/types.ts43-51NumericLiteral (9) to RegularExpressionLiteral (14). src/compiler/types.ts56-61BreakKeyword (84) to DeferKeyword (226). src/compiler/types.ts141-226Sources: src/compiler/types.ts40-492 src/compiler/scanner.ts1-100
The scan function is the core engine. It uses a switch statement on the character code at the current pos to determine the next token.
Core Scanning Algorithm:
skipTrivia is true, the scanner automatically skips whitespace and comments by calling skipWhiteSpace and skipComments. src/compiler/scanner.ts1210-1250text.charCodeAt(pos). src/compiler/scanner.ts1255-1260scanNumber. src/compiler/scanner.ts1000-1100scanString. src/compiler/scanner.ts800-950isKeyword or a lookup to see if it matches a reserved word. src/compiler/scanner.ts1300-1350tokenPos is set to the start of the token, and pos is advanced to the end. src/compiler/scanner.ts1205-1210Sources: src/compiler/scanner.ts1200-1800 src/compiler/scanner.ts800-1100
TokenFlags provide metadata about the scanned token, which is crucial for the parser (e.g., for Automatic Semicolon Insertion).
| Flag | Meaning |
|---|---|
PrecedingLineBreak | Set if there was a newline before this token. src/compiler/types.ts5552 |
Unterminated | Set if a string or template literal was not closed. src/compiler/types.ts5554 |
Scientific | Set for numeric literals using e notation. src/compiler/types.ts5556 |
HexSpecifier | Set for 0x hex literals. src/compiler/types.ts5558 |
Sources: src/compiler/types.ts5551-5564 src/compiler/scanner.ts637-640
The scanner must adapt its behavior based on the grammatical context provided by the parser.
When the parser enters a JSDoc comment, it uses scanJSDocToken. This mode recognizes tokens like @ (AtToken) and { (OpenBraceToken) differently than standard code.
Sources: src/compiler/scanner.ts1950-2100
In JSX mode (LanguageVariant.JSX), the scanner must handle < as the start of a tag rather than a "less than" operator. The parser calls scanJsxToken or scanJsxAttributeValue to handle these contexts.
Sources: src/compiler/scanner.ts1850-1950
Sometimes the parser realizes a token was scanned incorrectly due to ambiguity.
reScanSlashToken(): Re-scans a / as a RegularExpressionLiteral instead of a SlashToken. src/compiler/scanner.ts2150-2200reScanGreaterToken(): Re-scans > to handle >> or >>> in generic contexts. src/compiler/scanner.ts2100-2140Sources: src/compiler/scanner.ts2100-2300
Lexical errors are identified during the scanning process. When an error occurs (e.g., an unterminated string literal), the scanner:
Unterminated flag in tokenFlags. src/compiler/scanner.ts850-860onError callback if one was provided to createScanner. src/compiler/scanner.ts510-520Error messages are defined in diagnosticMessages.json. For example, "Unterminated string literal" has code 1002.
Sources: src/compiler/scanner.ts500-550 src/compiler/diagnosticMessages.json2-5
| Component | Responsibility | Code Reference |
|---|---|---|
CharacterCodes | Constants for char comparison (e.g., _, $, 0) | src/compiler/types.ts31-100 |
skipWhiteSpace | Advances pos past spaces, tabs, and BOMs | src/compiler/scanner.ts650-700 |
scanString | Handles escape sequences and quote matching | src/compiler/scanner.ts800-950 |
isKeyword | Fast check for reserved words | src/compiler/scanner.ts100-150 |
TokenFlags | Metadata for the current token | src/compiler/types.ts5551-5564 |
Sources: src/compiler/scanner.ts1-2387 src/compiler/types.ts1-5600
Refresh this wiki