Back to index

moin  1.9.0~rc2
Public Member Functions | Public Attributes | Static Public Attributes
MoinMoin.support.pygments.lexers.special.RawTokenLexer Class Reference
Inheritance diagram for MoinMoin.support.pygments.lexers.special.RawTokenLexer:
Inheritance graph
[legend]
Collaboration diagram for MoinMoin.support.pygments.lexers.special.RawTokenLexer:
Collaboration graph
[legend]

List of all members.

Public Member Functions

def __init__
def get_tokens
def get_tokens_unprocessed

Public Attributes

 compress

Static Public Attributes

string name = 'Raw token data'
list aliases = ['raw']
list filenames = []
list mimetypes = ['application/x-pygments-tokens']

Detailed Description

Recreate a token stream formatted with the `RawTokenFormatter`.  This
lexer raises exceptions during parsing if the token stream in the
file is malformed.

Additional options accepted:

`compress`
    If set to ``"gz"`` or ``"bz2"``, decompress the token stream with
    the given compression algorithm before lexing (default: ``""``).

Definition at line 40 of file special.py.


Constructor & Destructor Documentation

Definition at line 57 of file special.py.

00057 
00058     def __init__(self, **options):
00059         self.compress = get_choice_opt(options, 'compress',
00060                                        ['', 'none', 'gz', 'bz2'], '')
00061         Lexer.__init__(self, **options)


Member Function Documentation

Definition at line 62 of file special.py.

00062 
00063     def get_tokens(self, text):
00064         if isinstance(text, unicode):
00065             # raw token stream never has any non-ASCII characters
00066             text = text.encode('ascii')
00067         if self.compress == 'gz':
00068             import gzip
00069             gzipfile = gzip.GzipFile('', 'rb', 9, cStringIO.StringIO(text))
00070             text = gzipfile.read()
00071         elif self.compress == 'bz2':
00072             import bz2
00073             text = bz2.decompress(text)
00074 
00075         # do not call Lexer.get_tokens() because we do not want Unicode
00076         # decoding to occur, and stripping is not optional.
00077         text = text.strip(b('\n')) + b('\n')
00078         for i, t, v in self.get_tokens_unprocessed(text):
00079             yield t, v

Here is the call graph for this function:

Definition at line 80 of file special.py.

00080 
00081     def get_tokens_unprocessed(self, text):
00082         length = 0
00083         for match in line_re.finditer(text):
00084             try:
00085                 ttypestr, val = match.group().split(b('\t'), 1)
00086             except ValueError:
00087                 val = match.group().decode(self.encoding)
00088                 ttype = Error
00089             else:
00090                 ttype = _ttype_cache.get(ttypestr)
00091                 if not ttype:
00092                     ttype = Token
00093                     ttypes = ttypestr.split('.')[1:]
00094                     for ttype_ in ttypes:
00095                         if not ttype_ or not ttype_[0].isupper():
00096                             raise ValueError('malformed token name')
00097                         ttype = getattr(ttype, ttype_)
00098                     _ttype_cache[ttypestr] = ttype
00099                 val = val[2:-2].decode('unicode-escape')
00100             yield length, ttype, val
00101             length += len(val)

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Definition at line 53 of file special.py.

Definition at line 58 of file special.py.

Definition at line 54 of file special.py.

list MoinMoin.support.pygments.lexers.special.RawTokenLexer.mimetypes = ['application/x-pygments-tokens'] [static]

Definition at line 55 of file special.py.

Definition at line 52 of file special.py.


The documentation for this class was generated from the following file: