Back to index

python3.2  3.2.2
Classes | Functions | Variables
tokenize Namespace Reference


class  TokenInfo


def group
def any
def maybe


string __author__ = 'Ka-Ping Yee <>'
tuple __credits__
tuple cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
list __all__
int NL = 1
int ENCODING = 2
string Whitespace = r'[ \f\t]*'
string Comment = r'#[^\r\n]*'
tuple Ignore = Whitespace+any(r'\\\r?\n' + Whitespace)
string Name = r'\w+'
string Hexnumber = r'0[xX][0-9a-fA-F]+'
string Binnumber = r'0[bB][01]+'
string Octnumber = r'0[oO][0-7]+'
string Decnumber = r'(?:0+|[1-9][0-9]*)'
tuple Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
string Exponent = r'[eE][-+]?[0-9]+'
tuple Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+')
string Expfloat = r'[0-9]+'
tuple Floatnumber = group(Pointfloat, Expfloat)
tuple Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
tuple Number = group(Imagnumber, Floatnumber, Intnumber)
string Single = r"[^'\\]*(?:\\.[^'\\]*)*'"
string Double = r'[^"\\]*(?:\\.[^"\\]*)*"'
string Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"
string Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'

Detailed Description

Tokenization help for Python programs.

tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens.  It decodes the bytes according to PEP-0263 for
determining source file encoding.

It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF).  It generates 5-tuples with these

the token type (see
the token (a string)
the starting (row, column) indices of the token (a 2-tuple of ints)
the ending (row, column) indices of the token (a 2-tuple of ints)
the original line (string)

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators.  Additionally, all token lists start with an ENCODING token
which tells you which encoding was used to decode the bytes stream.

Function Documentation

def tokenize.any (   choices)

Definition at line 56 of file

def any(*choices): return group(*choices) + '*'
def (   choices)

Definition at line 55 of file

def group(*choices): return '(' + '|'.join(choices) + ')'
def tokenize.maybe (   choices)

Definition at line 57 of file

00058 def maybe(*choices): return group(*choices) + '?'
00060 # Note: we use unicode matching for names ("\w") but ascii matching for
# number literals.

Variable Documentation

Initial value:
00001 token.__all__+["COMMENT", "tokenize", "detect_encoding",
00002                            "NL", "untokenize", "ENCODING", "TokenInfo"]

Definition at line 37 of file

string tokenize.__author__ = 'Ka-Ping Yee <>'

Definition at line 23 of file

Initial value:
00001 ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
00002                'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
00003                'Michael Foord')

Definition at line 24 of file

string tokenize.Binnumber = r'0[bB][01]+'

Definition at line 67 of file

Definition at line 41 of file

Definition at line 62 of file

tuple tokenize.cookie_re = re.compile("coding[:=]\s*([-\w.]+)")

Definition at line 34 of file

string tokenize.Decnumber = r'(?:0+|[1-9][0-9]*)'

Definition at line 69 of file

string tokenize.Double = r'[^"\\]*(?:\\.[^"\\]*)*"'

Definition at line 81 of file

string tokenize.Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'

Definition at line 85 of file

Definition at line 45 of file

Definition at line 73 of file

string tokenize.Exponent = r'[eE][-+]?[0-9]+'

Definition at line 71 of file

Definition at line 74 of file

string tokenize.Hexnumber = r'0[xX][0-9a-fA-F]+'

Definition at line 66 of file

tuple tokenize.Ignore = Whitespace+any(r'\\\r?\n' + Whitespace)

Definition at line 63 of file

tuple tokenize.Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')

Definition at line 75 of file

Definition at line 70 of file

Definition at line 64 of file

Definition at line 43 of file

Definition at line 76 of file

string tokenize.Octnumber = r'0[oO][0-7]+'

Definition at line 68 of file

tuple tokenize.Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+')

Definition at line 72 of file

string tokenize.Single = r"[^'\\]*(?:\\.[^'\\]*)*'"

Definition at line 79 of file

string tokenize.Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"

Definition at line 83 of file

string tokenize.Whitespace = r'[ \f\t]*'

Definition at line 61 of file