Back to index

python3.2  3.2.2
Classes | Functions | Variables
tokenize Namespace Reference

Classes

class  TokenInfo

Functions

def group
def any
def maybe

Variables

string __author__ = 'Ka-Ping Yee <ping@lfw.org>'
tuple __credits__
tuple cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
list __all__
 COMMENT = N_TOKENS
int NL = 1
int ENCODING = 2
string Whitespace = r'[ \f\t]*'
string Comment = r'#[^\r\n]*'
tuple Ignore = Whitespace+any(r'\\\r?\n' + Whitespace)
string Name = r'\w+'
string Hexnumber = r'0[xX][0-9a-fA-F]+'
string Binnumber = r'0[bB][01]+'
string Octnumber = r'0[oO][0-7]+'
string Decnumber = r'(?:0+|[1-9][0-9]*)'
tuple Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
string Exponent = r'[eE][-+]?[0-9]+'
tuple Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+')
string Expfloat = r'[0-9]+'
tuple Floatnumber = group(Pointfloat, Expfloat)
tuple Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
tuple Number = group(Imagnumber, Floatnumber, Intnumber)
string Single = r"[^'\\]*(?:\\.[^'\\]*)*'"
string Double = r'[^"\\]*(?:\\.[^"\\]*)*"'
string Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"
string Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'

Detailed Description

Tokenization help for Python programs.

tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens.  It decodes the bytes according to PEP-0263 for
determining source file encoding.

It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF).  It generates 5-tuples with these
members:

the token type (see token.py)
the token (a string)
the starting (row, column) indices of the token (a 2-tuple of ints)
the ending (row, column) indices of the token (a 2-tuple of ints)
the original line (string)

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators.  Additionally, all token lists start with an ENCODING token
which tells you which encoding was used to decode the bytes stream.

Function Documentation

def tokenize.any (   choices)

Definition at line 56 of file tokenize.py.

00056 
def any(*choices): return group(*choices) + '*'
def tokenize.group (   choices)

Definition at line 55 of file tokenize.py.

00055 
def group(*choices): return '(' + '|'.join(choices) + ')'
def tokenize.maybe (   choices)

Definition at line 57 of file tokenize.py.

00057 
00058 def maybe(*choices): return group(*choices) + '?'
00059 
00060 # Note: we use unicode matching for names ("\w") but ascii matching for
# number literals.

Variable Documentation

Initial value:
00001 token.__all__+["COMMENT", "tokenize", "detect_encoding",
00002                            "NL", "untokenize", "ENCODING", "TokenInfo"]

Definition at line 37 of file tokenize.py.

string tokenize.__author__ = 'Ka-Ping Yee <ping@lfw.org>'

Definition at line 23 of file tokenize.py.

Initial value:
00001 ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
00002                'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
00003                'Michael Foord')

Definition at line 24 of file tokenize.py.

string tokenize.Binnumber = r'0[bB][01]+'

Definition at line 67 of file tokenize.py.

Definition at line 41 of file tokenize.py.

Definition at line 62 of file tokenize.py.

tuple tokenize.cookie_re = re.compile("coding[:=]\s*([-\w.]+)")

Definition at line 34 of file tokenize.py.

string tokenize.Decnumber = r'(?:0+|[1-9][0-9]*)'

Definition at line 69 of file tokenize.py.

string tokenize.Double = r'[^"\\]*(?:\\.[^"\\]*)*"'

Definition at line 81 of file tokenize.py.

string tokenize.Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'

Definition at line 85 of file tokenize.py.

Definition at line 45 of file tokenize.py.

Definition at line 73 of file tokenize.py.

string tokenize.Exponent = r'[eE][-+]?[0-9]+'

Definition at line 71 of file tokenize.py.

Definition at line 74 of file tokenize.py.

string tokenize.Hexnumber = r'0[xX][0-9a-fA-F]+'

Definition at line 66 of file tokenize.py.

tuple tokenize.Ignore = Whitespace+any(r'\\\r?\n' + Whitespace)

Definition at line 63 of file tokenize.py.

tuple tokenize.Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')

Definition at line 75 of file tokenize.py.

Definition at line 70 of file tokenize.py.

Definition at line 64 of file tokenize.py.

Definition at line 43 of file tokenize.py.

Definition at line 76 of file tokenize.py.

string tokenize.Octnumber = r'0[oO][0-7]+'

Definition at line 68 of file tokenize.py.

tuple tokenize.Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+')

Definition at line 72 of file tokenize.py.

string tokenize.Single = r"[^'\\]*(?:\\.[^'\\]*)*'"

Definition at line 79 of file tokenize.py.

string tokenize.Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"

Definition at line 83 of file tokenize.py.

string tokenize.Whitespace = r'[ \f\t]*'

Definition at line 61 of file tokenize.py.