Back to index

python3.2  3.2.2
Classes | Functions | Variables
test.test_tokenize Namespace Reference

Classes

class  TestTokenizerAdheresToPep0263
class  Test_Tokenize
class  TestDetectEncoding
class  TestTokenize

Functions

def dump_tokens
def roundtrip
def decistmt
def test_main

Variables

string doctests
dictionary __test__ = {"doctests" : doctests, 'decistmt': decistmt}

Function Documentation

Substitute Decimals for floats in a string of statements.

>>> from decimal import Decimal
>>> s = 'print(+21.3e-5*-.1234/81.7)'
>>> decistmt(s)
"print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"

The format of the exponent is inherited from the platform C library.
Known cases are "e-007" (Windows) and "e-07" (not Windows).  Since
we're only showing 11 digits, and the 12th isn't close to 5, the
rest of the output should be platform-independent.

>>> exec(s) #doctest: +ELLIPSIS
-3.2171603427...e-0...7

Output from calculations with Decimal should be identical across all
platforms.

>>> exec(decistmt(s))
-3.217160342717258261933904529E-7

Definition at line 608 of file test_tokenize.py.

00608 
00609 def decistmt(s):
00610     """Substitute Decimals for floats in a string of statements.
00611 
00612     >>> from decimal import Decimal
00613     >>> s = 'print(+21.3e-5*-.1234/81.7)'
00614     >>> decistmt(s)
00615     "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"
00616 
00617     The format of the exponent is inherited from the platform C library.
00618     Known cases are "e-007" (Windows) and "e-07" (not Windows).  Since
00619     we're only showing 11 digits, and the 12th isn't close to 5, the
00620     rest of the output should be platform-independent.
00621 
00622     >>> exec(s) #doctest: +ELLIPSIS
00623     -3.2171603427...e-0...7
00624 
00625     Output from calculations with Decimal should be identical across all
00626     platforms.
00627 
00628     >>> exec(decistmt(s))
00629     -3.217160342717258261933904529E-7
00630     """
00631     result = []
00632     g = tokenize(BytesIO(s.encode('utf-8')).readline)   # tokenize the string
00633     for toknum, tokval, _, _, _  in g:
00634         if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
00635             result.extend([
00636                 (NAME, 'Decimal'),
00637                 (OP, '('),
00638                 (STRING, repr(tokval)),
00639                 (OP, ')')
00640             ])
00641         else:
00642             result.append((toknum, tokval))
00643     return untokenize(result).decode('utf-8')
00644 

Here is the call graph for this function:

Print out the tokens in s in a table format.

The ENDMARKER is omitted.

Definition at line 576 of file test_tokenize.py.

00576 
00577 def dump_tokens(s):
00578     """Print out the tokens in s in a table format.
00579 
00580     The ENDMARKER is omitted.
00581     """
00582     f = BytesIO(s.encode('utf-8'))
00583     for type, token, start, end, line in tokenize(f.readline):
00584         if type == ENDMARKER:
00585             break
00586         type = tok_name[type]
00587         print("%(type)-10.10s %(token)-13.13r %(start)s %(end)s" % locals())

Test roundtrip for `untokenize`. `f` is an open file or a string.
The source code in f is tokenized, converted back to source code via
tokenize.untokenize(), and tokenized again from the latter. The test
fails if the second tokenization doesn't match the first.

Definition at line 588 of file test_tokenize.py.

00588 
00589 def roundtrip(f):
00590     """
00591     Test roundtrip for `untokenize`. `f` is an open file or a string.
00592     The source code in f is tokenized, converted back to source code via
00593     tokenize.untokenize(), and tokenized again from the latter. The test
00594     fails if the second tokenization doesn't match the first.
00595     """
00596     if isinstance(f, str):
00597         f = BytesIO(f.encode('utf-8'))
00598     try:
00599         token_list = list(tokenize(f.readline))
00600     finally:
00601         f.close()
00602     tokens1 = [tok[:2] for tok in token_list]
00603     new_bytes = untokenize(tokens1)
00604     readline = (line for line in new_bytes.splitlines(1)).__next__
00605     tokens2 = [tok[:2] for tok in tokenize(readline)]
00606     return tokens1 == tokens2
00607 
# This is an example from the docs, set up as a doctest.

Here is the caller graph for this function:

Definition at line 928 of file test_tokenize.py.

00928 
00929 def test_main():
00930     from test import test_tokenize
00931     support.run_doctest(test_tokenize, True)
00932     support.run_unittest(TestTokenizerAdheresToPep0263)
00933     support.run_unittest(Test_Tokenize)
00934     support.run_unittest(TestDetectEncoding)
00935     support.run_unittest(TestTokenize)


Variable Documentation

Definition at line 926 of file test_tokenize.py.

Definition at line 1 of file test_tokenize.py.