Back to index

python3.2  3.2.2
Public Member Functions | Public Attributes
lib2to3.pgen2.conv.Converter Class Reference
Inheritance diagram for lib2to3.pgen2.conv.Converter:
Inheritance graph
[legend]
Collaboration diagram for lib2to3.pgen2.conv.Converter:
Collaboration graph
[legend]

List of all members.

Public Member Functions

def run
def parse_graminit_h
def parse_graminit_c
def finish_off
def dump
def load
def copy
def report

Public Attributes

 symbol2number
 number2symbol
 states
 dfas
 labels
 start
 keywords
 tokens
 symbol2label
_PyObject_HEAD_EXTRA Py_ssize_t ob_refcnt
struct _typeobjectob_type

Detailed Description

Grammar subclass that reads classic pgen output files.

The run() method reads the tables as produced by the pgen parser
generator, typically contained in two C files, graminit.h and
graminit.c.  The other methods are for internal use only.

See the base class for more documentation.

Definition at line 36 of file conv.py.


Member Function Documentation

def lib2to3.pgen2.grammar.Grammar.copy (   self) [inherited]
Copy the grammar.

Definition at line 100 of file grammar.py.

00100 
00101     def copy(self):
00102         """
00103         Copy the grammar.
00104         """
00105         new = self.__class__()
00106         for dict_attr in ("symbol2number", "number2symbol", "dfas", "keywords",
00107                           "tokens", "symbol2label"):
00108             setattr(new, dict_attr, getattr(self, dict_attr).copy())
00109         new.labels = self.labels[:]
00110         new.states = self.states[:]
00111         new.start = self.start
00112         return new

Here is the call graph for this function:

def lib2to3.pgen2.grammar.Grammar.dump (   self,
  filename 
) [inherited]
Dump the grammar tables to a pickle file.

Definition at line 87 of file grammar.py.

00087 
00088     def dump(self, filename):
00089         """Dump the grammar tables to a pickle file."""
00090         f = open(filename, "wb")
00091         pickle.dump(self.__dict__, f, 2)
00092         f.close()

Here is the call graph for this function:

Create additional useful structures.  (Internal).

Definition at line 249 of file conv.py.

00249 
00250     def finish_off(self):
00251         """Create additional useful structures.  (Internal)."""
00252         self.keywords = {} # map from keyword strings to arc labels
00253         self.tokens = {}   # map from numeric token values to arc labels
00254         for ilabel, (type, value) in enumerate(self.labels):
00255             if type == token.NAME and value is not None:
00256                 self.keywords[value] = ilabel
00257             elif value is None:
00258                 self.tokens[type] = ilabel

Here is the caller graph for this function:

def lib2to3.pgen2.grammar.Grammar.load (   self,
  filename 
) [inherited]
Load the grammar tables from a pickle file.

Definition at line 93 of file grammar.py.

00093 
00094     def load(self, filename):
00095         """Load the grammar tables from a pickle file."""
00096         f = open(filename, "rb")
00097         d = pickle.load(f)
00098         f.close()
00099         self.__dict__.update(d)

Here is the call graph for this function:

def lib2to3.pgen2.conv.Converter.parse_graminit_c (   self,
  filename 
)
Parse the .c file written by pgen.  (Internal)

The file looks as follows.  The first two lines are always this:

#include "pgenheaders.h"
#include "grammar.h"

After that come four blocks:

1) one or more state definitions
2) a table defining dfas
3) a table defining labels
4) a struct defining the grammar

A state definition has the following form:
- one or more arc arrays, each of the form:
  static arc arcs_<n>_<m>[<k>] = {
  {<i>, <j>},
  ...
  };
- followed by a state array, of the form:
  static state states_<s>[<t>] = {
  {<k>, arcs_<n>_<m>},
  ...
  };

Definition at line 84 of file conv.py.

00084 
00085     def parse_graminit_c(self, filename):
00086         """Parse the .c file written by pgen.  (Internal)
00087 
00088         The file looks as follows.  The first two lines are always this:
00089 
00090         #include "pgenheaders.h"
00091         #include "grammar.h"
00092 
00093         After that come four blocks:
00094 
00095         1) one or more state definitions
00096         2) a table defining dfas
00097         3) a table defining labels
00098         4) a struct defining the grammar
00099 
00100         A state definition has the following form:
00101         - one or more arc arrays, each of the form:
00102           static arc arcs_<n>_<m>[<k>] = {
00103                   {<i>, <j>},
00104                   ...
00105           };
00106         - followed by a state array, of the form:
00107           static state states_<s>[<t>] = {
00108                   {<k>, arcs_<n>_<m>},
00109                   ...
00110           };
00111 
00112         """
00113         try:
00114             f = open(filename)
00115         except IOError as err:
00116             print("Can't open %s: %s" % (filename, err))
00117             return False
00118         # The code below essentially uses f's iterator-ness!
00119         lineno = 0
00120 
00121         # Expect the two #include lines
00122         lineno, line = lineno+1, next(f)
00123         assert line == '#include "pgenheaders.h"\n', (lineno, line)
00124         lineno, line = lineno+1, next(f)
00125         assert line == '#include "grammar.h"\n', (lineno, line)
00126 
00127         # Parse the state definitions
00128         lineno, line = lineno+1, next(f)
00129         allarcs = {}
00130         states = []
00131         while line.startswith("static arc "):
00132             while line.startswith("static arc "):
00133                 mo = re.match(r"static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$",
00134                               line)
00135                 assert mo, (lineno, line)
00136                 n, m, k = list(map(int, mo.groups()))
00137                 arcs = []
00138                 for _ in range(k):
00139                     lineno, line = lineno+1, next(f)
00140                     mo = re.match(r"\s+{(\d+), (\d+)},$", line)
00141                     assert mo, (lineno, line)
00142                     i, j = list(map(int, mo.groups()))
00143                     arcs.append((i, j))
00144                 lineno, line = lineno+1, next(f)
00145                 assert line == "};\n", (lineno, line)
00146                 allarcs[(n, m)] = arcs
00147                 lineno, line = lineno+1, next(f)
00148             mo = re.match(r"static state states_(\d+)\[(\d+)\] = {$", line)
00149             assert mo, (lineno, line)
00150             s, t = list(map(int, mo.groups()))
00151             assert s == len(states), (lineno, line)
00152             state = []
00153             for _ in range(t):
00154                 lineno, line = lineno+1, next(f)
00155                 mo = re.match(r"\s+{(\d+), arcs_(\d+)_(\d+)},$", line)
00156                 assert mo, (lineno, line)
00157                 k, n, m = list(map(int, mo.groups()))
00158                 arcs = allarcs[n, m]
00159                 assert k == len(arcs), (lineno, line)
00160                 state.append(arcs)
00161             states.append(state)
00162             lineno, line = lineno+1, next(f)
00163             assert line == "};\n", (lineno, line)
00164             lineno, line = lineno+1, next(f)
00165         self.states = states
00166 
00167         # Parse the dfas
00168         dfas = {}
00169         mo = re.match(r"static dfa dfas\[(\d+)\] = {$", line)
00170         assert mo, (lineno, line)
00171         ndfas = int(mo.group(1))
00172         for i in range(ndfas):
00173             lineno, line = lineno+1, next(f)
00174             mo = re.match(r'\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$',
00175                           line)
00176             assert mo, (lineno, line)
00177             symbol = mo.group(2)
00178             number, x, y, z = list(map(int, mo.group(1, 3, 4, 5)))
00179             assert self.symbol2number[symbol] == number, (lineno, line)
00180             assert self.number2symbol[number] == symbol, (lineno, line)
00181             assert x == 0, (lineno, line)
00182             state = states[z]
00183             assert y == len(state), (lineno, line)
00184             lineno, line = lineno+1, next(f)
00185             mo = re.match(r'\s+("(?:\\\d\d\d)*")},$', line)
00186             assert mo, (lineno, line)
00187             first = {}
00188             rawbitset = eval(mo.group(1))
00189             for i, c in enumerate(rawbitset):
00190                 byte = ord(c)
00191                 for j in range(8):
00192                     if byte & (1<<j):
00193                         first[i*8 + j] = 1
00194             dfas[number] = (state, first)
00195         lineno, line = lineno+1, next(f)
00196         assert line == "};\n", (lineno, line)
00197         self.dfas = dfas
00198 
00199         # Parse the labels
00200         labels = []
00201         lineno, line = lineno+1, next(f)
00202         mo = re.match(r"static label labels\[(\d+)\] = {$", line)
00203         assert mo, (lineno, line)
00204         nlabels = int(mo.group(1))
00205         for i in range(nlabels):
00206             lineno, line = lineno+1, next(f)
00207             mo = re.match(r'\s+{(\d+), (0|"\w+")},$', line)
00208             assert mo, (lineno, line)
00209             x, y = mo.groups()
00210             x = int(x)
00211             if y == "0":
00212                 y = None
00213             else:
00214                 y = eval(y)
00215             labels.append((x, y))
00216         lineno, line = lineno+1, next(f)
00217         assert line == "};\n", (lineno, line)
00218         self.labels = labels
00219 
00220         # Parse the grammar struct
00221         lineno, line = lineno+1, next(f)
00222         assert line == "grammar _PyParser_Grammar = {\n", (lineno, line)
00223         lineno, line = lineno+1, next(f)
00224         mo = re.match(r"\s+(\d+),$", line)
00225         assert mo, (lineno, line)
00226         ndfas = int(mo.group(1))
00227         assert ndfas == len(self.dfas)
00228         lineno, line = lineno+1, next(f)
00229         assert line == "\tdfas,\n", (lineno, line)
00230         lineno, line = lineno+1, next(f)
00231         mo = re.match(r"\s+{(\d+), labels},$", line)
00232         assert mo, (lineno, line)
00233         nlabels = int(mo.group(1))
00234         assert nlabels == len(self.labels), (lineno, line)
00235         lineno, line = lineno+1, next(f)
00236         mo = re.match(r"\s+(\d+)$", line)
00237         assert mo, (lineno, line)
00238         start = int(mo.group(1))
00239         assert start in self.number2symbol, (lineno, line)
00240         self.start = start
00241         lineno, line = lineno+1, next(f)
00242         assert line == "};\n", (lineno, line)
00243         try:
00244             lineno, line = lineno+1, next(f)
00245         except StopIteration:
00246             pass
00247         else:
00248             assert 0, (lineno, line)

Here is the call graph for this function:

Here is the caller graph for this function:

def lib2to3.pgen2.conv.Converter.parse_graminit_h (   self,
  filename 
)
Parse the .h file written by pgen.  (Internal)

This file is a sequence of #define statements defining the
nonterminals of the grammar as numbers.  We build two tables
mapping the numbers to names and back.

Definition at line 53 of file conv.py.

00053 
00054     def parse_graminit_h(self, filename):
00055         """Parse the .h file written by pgen.  (Internal)
00056 
00057         This file is a sequence of #define statements defining the
00058         nonterminals of the grammar as numbers.  We build two tables
00059         mapping the numbers to names and back.
00060 
00061         """
00062         try:
00063             f = open(filename)
00064         except IOError as err:
00065             print("Can't open %s: %s" % (filename, err))
00066             return False
00067         self.symbol2number = {}
00068         self.number2symbol = {}
00069         lineno = 0
00070         for line in f:
00071             lineno += 1
00072             mo = re.match(r"^#define\s+(\w+)\s+(\d+)$", line)
00073             if not mo and line.strip():
00074                 print("%s(%s): can't parse %s" % (filename, lineno,
00075                                                   line.strip()))
00076             else:
00077                 symbol, number = mo.groups()
00078                 number = int(number)
00079                 assert symbol not in self.symbol2number
00080                 assert number not in self.number2symbol
00081                 self.symbol2number[symbol] = number
00082                 self.number2symbol[number] = symbol
00083         return True

Here is the caller graph for this function:

def lib2to3.pgen2.grammar.Grammar.report (   self) [inherited]
Dump the grammar tables to standard output, for debugging.

Definition at line 113 of file grammar.py.

00113 
00114     def report(self):
00115         """Dump the grammar tables to standard output, for debugging."""
00116         from pprint import pprint
00117         print("s2n")
00118         pprint(self.symbol2number)
00119         print("n2s")
00120         pprint(self.number2symbol)
00121         print("states")
00122         pprint(self.states)
00123         print("dfas")
00124         pprint(self.dfas)
00125         print("labels")
00126         pprint(self.labels)
00127         print("start", self.start)
00128 
00129 
00130 # Map from operator to number (since tokenize doesn't do this)

Here is the caller graph for this function:

def lib2to3.pgen2.conv.Converter.run (   self,
  graminit_h,
  graminit_c 
)
Load the grammar tables from the text files written by pgen.

Definition at line 47 of file conv.py.

00047 
00048     def run(self, graminit_h, graminit_c):
00049         """Load the grammar tables from the text files written by pgen."""
00050         self.parse_graminit_h(graminit_h)
00051         self.parse_graminit_c(graminit_c)
00052         self.finish_off()

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 196 of file conv.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 251 of file conv.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 217 of file conv.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 67 of file conv.py.

Definition at line 107 of file object.h.

struct _typeobject* _object::ob_type [inherited]

Definition at line 108 of file object.h.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 239 of file conv.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 164 of file conv.py.

Definition at line 84 of file grammar.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 66 of file conv.py.

Reimplemented from lib2to3.pgen2.grammar.Grammar.

Definition at line 252 of file conv.py.


The documentation for this class was generated from the following file: