lexical analyzer

Definitions

A component in a compiler that converts a sequence of characters into a sequence of tokens.

A program or function that performs lexical analysis by breaking source code into meaningful units called lexemes.

Examples

The lexical analyzer breaks the source code into a stream of tokens for the parser to process.

During the compilation process, the lexical analyzer identifies keywords, identifiers, and operators.

A common tool used to generate a lexical analyzer automatically is Lex or Flex.

Synonyms

scanner tokenizer lexer lexical scanner lexical tokenizer

Antonyms

parser syntactic analyzer syntax analyzer code generator semantic analyzer