Skip to content

Latest commit

 

History

History
2 lines (2 loc) · 451 Bytes

README.md

File metadata and controls

2 lines (2 loc) · 451 Bytes

lexical_analyzer

In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters into a sequence of lexical tokens. A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, although scanner is also a term for the first stage of a lexer. A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.