Lexical Analysis

• Lexical Analysis

http://www.youtube.com/watch?v=B-nkdR4H530
http://www.youtube.com/watch?v=_1o9z4XIyx4

• Lexical analysis

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens
A program or function which performs lexical analysis is called a lexical analyzer, lexer, or scanner.
A lexer often exists as a single function which is called by a parser or another function.

http://en.wikipedia.org/wiki/Lexical_analysis

• compiler phases

• Lexical Analysis

A scanner groups input characters into tokens. For example, if the input is

x = x*(b+1);

then the scanner generates the following sequence of tokens

id(x)

=

id(x)

*

(

id(b)

+

num(1)

)

;

A typical scanner:

recognizes the keywords of the language (these are the reserved words that have a special meaning in the language, such as the word class in Java);

recognizes special characters, such as ( and ), or groups of special characters, such as := and ==;

recognizes identifiers, integers, reals, decimals, strings, etc;

ignores whitespaces (tabs and blanks) and comments;

recognizes and processes special directives (such as the #include "file" directive in C) and macros.

http://lambda.uta.edu/cse5317/notes/node6.html