Back to index

python3.2  3.2.2
Public Member Functions | Public Attributes | Static Public Attributes | Private Member Functions | Static Private Attributes
pickle._Pickler Class Reference

List of all members.

Public Member Functions

def __init__
def clear_memo
def dump
def memoize
def put
def get
def save
def persistent_id
def save_pers
def save_reduce
def save_none
def save_bool
def save_long
def save_float
def save_bytes
def save_str
def save_tuple
def save_list
def save_dict
def save_global

Public Attributes

 write
 memo
 proto
 bin
 fast
 fix_imports

Static Public Attributes

dictionary dispatch = {}

Private Member Functions

def _batch_appends
def _batch_setitems

Static Private Attributes

int _BATCHSIZE = 1000

Detailed Description

Definition at line 177 of file pickle.py.


Constructor & Destructor Documentation

def pickle._Pickler.__init__ (   self,
  file,
  protocol = None,
  fix_imports = True 
)
This takes a binary file for writing a pickle data stream.

The optional protocol argument tells the pickler to use the
given protocol; supported protocols are 0, 1, 2, 3.  The default
protocol is 3; a backward-incompatible protocol designed for
Python 3.0.

Specifying a negative protocol version selects the highest
protocol version supported.  The higher the protocol used, the
more recent the version of Python needed to read the pickle
produced.

The file argument must have a write() method that accepts a single
bytes argument. It can thus be a file object opened for binary
writing, a io.BytesIO instance, or any other custom object that
meets this interface.

If fix_imports is True and protocol is less than 3, pickle will try to
map the new Python 3.x names to the old module names used in Python
2.x, so that the pickle data stream is readable with Python 2.x.

Definition at line 179 of file pickle.py.

00179 
00180     def __init__(self, file, protocol=None, *, fix_imports=True):
00181         """This takes a binary file for writing a pickle data stream.
00182 
00183         The optional protocol argument tells the pickler to use the
00184         given protocol; supported protocols are 0, 1, 2, 3.  The default
00185         protocol is 3; a backward-incompatible protocol designed for
00186         Python 3.0.
00187 
00188         Specifying a negative protocol version selects the highest
00189         protocol version supported.  The higher the protocol used, the
00190         more recent the version of Python needed to read the pickle
00191         produced.
00192 
00193         The file argument must have a write() method that accepts a single
00194         bytes argument. It can thus be a file object opened for binary
00195         writing, a io.BytesIO instance, or any other custom object that
00196         meets this interface.
00197 
00198         If fix_imports is True and protocol is less than 3, pickle will try to
00199         map the new Python 3.x names to the old module names used in Python
00200         2.x, so that the pickle data stream is readable with Python 2.x.
00201         """
00202         if protocol is None:
00203             protocol = DEFAULT_PROTOCOL
00204         if protocol < 0:
00205             protocol = HIGHEST_PROTOCOL
00206         elif not 0 <= protocol <= HIGHEST_PROTOCOL:
00207             raise ValueError("pickle protocol must be <= %d" % HIGHEST_PROTOCOL)
00208         try:
00209             self.write = file.write
00210         except AttributeError:
00211             raise TypeError("file must have a 'write' attribute")
00212         self.memo = {}
00213         self.proto = int(protocol)
00214         self.bin = protocol >= 1
00215         self.fast = 0
00216         self.fix_imports = fix_imports and protocol < 3

Here is the caller graph for this function:


Member Function Documentation

def pickle._Pickler._batch_appends (   self,
  items 
) [private]

Definition at line 581 of file pickle.py.

00581 
00582     def _batch_appends(self, items):
00583         # Helper to batch up APPENDS sequences
00584         save = self.save
00585         write = self.write
00586 
00587         if not self.bin:
00588             for x in items:
00589                 save(x)
00590                 write(APPEND)
00591             return
00592 
00593         items = iter(items)
00594         r = range(self._BATCHSIZE)
00595         while items is not None:
00596             tmp = []
00597             for i in r:
00598                 try:
00599                     x = next(items)
00600                     tmp.append(x)
00601                 except StopIteration:
00602                     items = None
00603                     break
00604             n = len(tmp)
00605             if n > 1:
00606                 write(MARK)
00607                 for x in tmp:
00608                     save(x)
00609                 write(APPENDS)
00610             elif n:
00611                 save(tmp[0])
00612                 write(APPEND)
00613             # else tmp is empty, and we're done

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler._batch_setitems (   self,
  items 
) [private]

Definition at line 629 of file pickle.py.

00629 
00630     def _batch_setitems(self, items):
00631         # Helper to batch up SETITEMS sequences; proto >= 1 only
00632         save = self.save
00633         write = self.write
00634 
00635         if not self.bin:
00636             for k, v in items:
00637                 save(k)
00638                 save(v)
00639                 write(SETITEM)
00640             return
00641 
00642         items = iter(items)
00643         r = range(self._BATCHSIZE)
00644         while items is not None:
00645             tmp = []
00646             for i in r:
00647                 try:
00648                     tmp.append(next(items))
00649                 except StopIteration:
00650                     items = None
00651                     break
00652             n = len(tmp)
00653             if n > 1:
00654                 write(MARK)
00655                 for k, v in tmp:
00656                     save(k)
00657                     save(v)
00658                 write(SETITEMS)
00659             elif n:
00660                 k, v = tmp[0]
00661                 save(k)
00662                 save(v)
00663                 write(SETITEM)
00664             # else tmp is empty, and we're done

Here is the call graph for this function:

Here is the caller graph for this function:

Clears the pickler's "memo".

The memo is the data structure that remembers which objects the
pickler has already seen, so that shared or recursive objects are
pickled by reference and not by value.  This method is useful when
re-using picklers.

Definition at line 217 of file pickle.py.

00217 
00218     def clear_memo(self):
00219         """Clears the pickler's "memo".
00220 
00221         The memo is the data structure that remembers which objects the
00222         pickler has already seen, so that shared or recursive objects are
00223         pickled by reference and not by value.  This method is useful when
00224         re-using picklers.
00225 
00226         """
00227         self.memo.clear()

def pickle._Pickler.dump (   self,
  obj 
)
Write a pickled representation of obj to the open file.

Definition at line 228 of file pickle.py.

00228 
00229     def dump(self, obj):
00230         """Write a pickled representation of obj to the open file."""
00231         # Check whether Pickler was initialized correctly. This is
00232         # only needed to mimic the behavior of _pickle.Pickler.dump().
00233         if not hasattr(self, "write"):
00234             raise PicklingError("Pickler.__init__() was not called by "
00235                                 "%s.__init__()" % (self.__class__.__name__,))
00236         if self.proto >= 2:
00237             self.write(PROTO + bytes([self.proto]))
00238         self.save(obj)
00239         self.write(STOP)

Here is the call graph for this function:

def pickle._Pickler.get (   self,
  i,
  pack = struct.pack 
)

Definition at line 273 of file pickle.py.

00273 
00274     def get(self, i, pack=struct.pack):
00275         if self.bin:
00276             if i < 256:
00277                 return BINGET + bytes([i])
00278             else:
00279                 return LONG_BINGET + pack("<i", i)
00280 
00281         return GET + repr(i).encode("ascii") + b'\n'

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.memoize (   self,
  obj 
)
Store an object in the memo.

Definition at line 240 of file pickle.py.

00240 
00241     def memoize(self, obj):
00242         """Store an object in the memo."""
00243 
00244         # The Pickler memo is a dictionary mapping object ids to 2-tuples
00245         # that contain the Unpickler memo key and the object being memoized.
00246         # The memo key is written to the pickle and will become
00247         # the key in the Unpickler's memo.  The object is stored in the
00248         # Pickler memo so that transient objects are kept alive during
00249         # pickling.
00250 
00251         # The use of the Unpickler memo length as the memo key is just a
00252         # convention.  The only requirement is that the memo values be unique.
00253         # But there appears no advantage to any other scheme, and this
00254         # scheme allows the Unpickler memo to be implemented as a plain (but
00255         # growable) array, indexed by memo key.
00256         if self.fast:
00257             return
00258         assert id(obj) not in self.memo
00259         memo_len = len(self.memo)
00260         self.write(self.put(memo_len))
00261         self.memo[id(obj)] = memo_len, obj

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.persistent_id (   self,
  obj 
)

Definition at line 346 of file pickle.py.

00346 
00347     def persistent_id(self, obj):
00348         # This exists so a subclass can override it
00349         return None

Here is the caller graph for this function:

def pickle._Pickler.put (   self,
  i,
  pack = struct.pack 
)

Definition at line 263 of file pickle.py.

00263 
00264     def put(self, i, pack=struct.pack):
00265         if self.bin:
00266             if i < 256:
00267                 return BINPUT + bytes([i])
00268             else:
00269                 return LONG_BINPUT + pack("<i", i)
00270 
00271         return PUT + repr(i).encode("ascii") + b'\n'

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.save (   self,
  obj,
  save_persistent_id = True 
)

Definition at line 282 of file pickle.py.

00282 
00283     def save(self, obj, save_persistent_id=True):
00284         # Check for persistent id (defined by a subclass)
00285         pid = self.persistent_id(obj)
00286         if pid is not None and save_persistent_id:
00287             self.save_pers(pid)
00288             return
00289 
00290         # Check the memo
00291         x = self.memo.get(id(obj))
00292         if x:
00293             self.write(self.get(x[0]))
00294             return
00295 
00296         # Check the type dispatch table
00297         t = type(obj)
00298         f = self.dispatch.get(t)
00299         if f:
00300             f(self, obj) # Call unbound method with explicit self
00301             return
00302 
00303         # Check for a class with a custom metaclass; treat as regular class
00304         try:
00305             issc = issubclass(t, type)
00306         except TypeError: # t is not a class (old Boost; see SF #502085)
00307             issc = 0
00308         if issc:
00309             self.save_global(obj)
00310             return
00311 
00312         # Check copyreg.dispatch_table
00313         reduce = dispatch_table.get(t)
00314         if reduce:
00315             rv = reduce(obj)
00316         else:
00317             # Check for a __reduce_ex__ method, fall back to __reduce__
00318             reduce = getattr(obj, "__reduce_ex__", None)
00319             if reduce:
00320                 rv = reduce(self.proto)
00321             else:
00322                 reduce = getattr(obj, "__reduce__", None)
00323                 if reduce:
00324                     rv = reduce()
00325                 else:
00326                     raise PicklingError("Can't pickle %r object: %r" %
00327                                         (t.__name__, obj))
00328 
00329         # Check for string returned by reduce(), meaning "save as global"
00330         if isinstance(rv, str):
00331             self.save_global(obj, rv)
00332             return
00333 
00334         # Assert that reduce() returned a tuple
00335         if not isinstance(rv, tuple):
00336             raise PicklingError("%s must return string or tuple" % reduce)
00337 
00338         # Assert that it returned an appropriately sized tuple
00339         l = len(rv)
00340         if not (2 <= l <= 5):
00341             raise PicklingError("Tuple returned by %s must have "
00342                                 "two to five elements" % reduce)
00343 
00344         # Save the reduce() output and finally memoize the object
00345         self.save_reduce(obj=obj, *rv)

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.save_bool (   self,
  obj 
)

Definition at line 443 of file pickle.py.

00443 
00444     def save_bool(self, obj):
00445         if self.proto >= 2:
00446             self.write(obj and NEWTRUE or NEWFALSE)
00447         else:
            self.write(obj and TRUE or FALSE)
def pickle._Pickler.save_bytes (   self,
  obj,
  pack = struct.pack 
)

Definition at line 488 of file pickle.py.

00488 
00489     def save_bytes(self, obj, pack=struct.pack):
00490         if self.proto < 3:
00491             self.save_reduce(bytes, (list(obj),), obj=obj)
00492             return
00493         n = len(obj)
00494         if n < 256:
00495             self.write(SHORT_BINBYTES + bytes([n]) + bytes(obj))
00496         else:
00497             self.write(BINBYTES + pack("<i", n) + bytes(obj))
        self.memoize(obj)

Here is the call graph for this function:

def pickle._Pickler.save_dict (   self,
  obj 
)

Definition at line 614 of file pickle.py.

00614 
00615     def save_dict(self, obj):
00616         write = self.write
00617 
00618         if self.bin:
00619             write(EMPTY_DICT)
00620         else:   # proto 0 -- can't use EMPTY_DICT
00621             write(MARK + DICT)
00622 
00623         self.memoize(obj)
00624         self._batch_setitems(obj.items())

Here is the call graph for this function:

def pickle._Pickler.save_float (   self,
  obj,
  pack = struct.pack 
)

Definition at line 481 of file pickle.py.

00481 
00482     def save_float(self, obj, pack=struct.pack):
00483         if self.bin:
00484             self.write(BINFLOAT + pack('>d', obj))
00485         else:
            self.write(FLOAT + repr(obj).encode("ascii") + b'\n')

Here is the call graph for this function:

def pickle._Pickler.save_global (   self,
  obj,
  name = None,
  pack = struct.pack 
)

Definition at line 665 of file pickle.py.

00665 
00666     def save_global(self, obj, name=None, pack=struct.pack):
00667         write = self.write
00668         memo = self.memo
00669 
00670         if name is None:
00671             name = obj.__name__
00672 
00673         module = getattr(obj, "__module__", None)
00674         if module is None:
00675             module = whichmodule(obj, name)
00676 
00677         try:
00678             __import__(module, level=0)
00679             mod = sys.modules[module]
00680             klass = getattr(mod, name)
00681         except (ImportError, KeyError, AttributeError):
00682             raise PicklingError(
00683                 "Can't pickle %r: it's not found as %s.%s" %
00684                 (obj, module, name))
00685         else:
00686             if klass is not obj:
00687                 raise PicklingError(
00688                     "Can't pickle %r: it's not the same object as %s.%s" %
00689                     (obj, module, name))
00690 
00691         if self.proto >= 2:
00692             code = _extension_registry.get((module, name))
00693             if code:
00694                 assert code > 0
00695                 if code <= 0xff:
00696                     write(EXT1 + bytes([code]))
00697                 elif code <= 0xffff:
00698                     write(EXT2 + bytes([code&0xff, code>>8]))
00699                 else:
00700                     write(EXT4 + pack("<i", code))
00701                 return
00702         # Non-ASCII identifiers are supported only with protocols >= 3.
00703         if self.proto >= 3:
00704             write(GLOBAL + bytes(module, "utf-8") + b'\n' +
00705                   bytes(name, "utf-8") + b'\n')
00706         else:
00707             if self.fix_imports:
00708                 if (module, name) in _compat_pickle.REVERSE_NAME_MAPPING:
00709                     module, name = _compat_pickle.REVERSE_NAME_MAPPING[(module, name)]
00710                 if module in _compat_pickle.REVERSE_IMPORT_MAPPING:
00711                     module = _compat_pickle.REVERSE_IMPORT_MAPPING[module]
00712             try:
00713                 write(GLOBAL + bytes(module, "ascii") + b'\n' +
00714                       bytes(name, "ascii") + b'\n')
00715             except UnicodeEncodeError:
00716                 raise PicklingError(
00717                     "can't pickle global identifier '%s.%s' using "
00718                     "pickle protocol %i" % (module, name, self.proto))
00719 
00720         self.memoize(obj)

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.save_list (   self,
  obj 
)

Definition at line 566 of file pickle.py.

00566 
00567     def save_list(self, obj):
00568         write = self.write
00569 
00570         if self.bin:
00571             write(EMPTY_LIST)
00572         else:   # proto 0 -- can't use EMPTY_LIST
00573             write(MARK + LIST)
00574 
00575         self.memoize(obj)
00576         self._batch_appends(obj)

Here is the call graph for this function:

def pickle._Pickler.save_long (   self,
  obj,
  pack = struct.pack 
)

Definition at line 450 of file pickle.py.

00450 
00451     def save_long(self, obj, pack=struct.pack):
00452         if self.bin:
00453             # If the int is small enough to fit in a signed 4-byte 2's-comp
00454             # format, we can store it more efficiently than the general
00455             # case.
00456             # First one- and two-byte unsigned ints:
00457             if obj >= 0:
00458                 if obj <= 0xff:
00459                     self.write(BININT1 + bytes([obj]))
00460                     return
00461                 if obj <= 0xffff:
00462                     self.write(BININT2 + bytes([obj&0xff, obj>>8]))
00463                     return
00464             # Next check for 4-byte signed ints:
00465             high_bits = obj >> 31  # note that Python shift sign-extends
00466             if high_bits == 0 or high_bits == -1:
00467                 # All high bits are copies of bit 2**31, so the value
00468                 # fits in a 4-byte signed int.
00469                 self.write(BININT + pack("<i", obj))
00470                 return
00471         if self.proto >= 2:
00472             encoded = encode_long(obj)
00473             n = len(encoded)
00474             if n < 256:
00475                 self.write(LONG1 + bytes([n]) + encoded)
00476             else:
00477                 self.write(LONG4 + pack("<i", n) + encoded)
00478             return
        self.write(LONG + repr(obj).encode("ascii") + b'L\n')

Here is the call graph for this function:

def pickle._Pickler.save_none (   self,
  obj 
)

Definition at line 439 of file pickle.py.

00439 
00440     def save_none(self, obj):
        self.write(NONE)
def pickle._Pickler.save_pers (   self,
  pid 
)

Definition at line 350 of file pickle.py.

00350 
00351     def save_pers(self, pid):
00352         # Save a persistent id reference
00353         if self.bin:
00354             self.save(pid, save_persistent_id=False)
00355             self.write(BINPERSID)
00356         else:
00357             self.write(PERSID + str(pid).encode("ascii") + b'\n')

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.save_reduce (   self,
  func,
  args,
  state = None,
  listitems = None,
  dictitems = None,
  obj = None 
)

Definition at line 359 of file pickle.py.

00359 
00360                     listitems=None, dictitems=None, obj=None):
00361         # This API is called by some subclasses
00362 
00363         # Assert that args is a tuple
00364         if not isinstance(args, tuple):
00365             raise PicklingError("args from save_reduce() should be a tuple")
00366 
00367         # Assert that func is callable
00368         if not hasattr(func, '__call__'):
00369             raise PicklingError("func from save_reduce() should be callable")
00370 
00371         save = self.save
00372         write = self.write
00373 
00374         # Protocol 2 special case: if func's name is __newobj__, use NEWOBJ
00375         if self.proto >= 2 and getattr(func, "__name__", "") == "__newobj__":
00376             # A __reduce__ implementation can direct protocol 2 to
00377             # use the more efficient NEWOBJ opcode, while still
00378             # allowing protocol 0 and 1 to work normally.  For this to
00379             # work, the function returned by __reduce__ should be
00380             # called __newobj__, and its first argument should be a
00381             # new-style class.  The implementation for __newobj__
00382             # should be as follows, although pickle has no way to
00383             # verify this:
00384             #
00385             # def __newobj__(cls, *args):
00386             #     return cls.__new__(cls, *args)
00387             #
00388             # Protocols 0 and 1 will pickle a reference to __newobj__,
00389             # while protocol 2 (and above) will pickle a reference to
00390             # cls, the remaining args tuple, and the NEWOBJ code,
00391             # which calls cls.__new__(cls, *args) at unpickling time
00392             # (see load_newobj below).  If __reduce__ returns a
00393             # three-tuple, the state from the third tuple item will be
00394             # pickled regardless of the protocol, calling __setstate__
00395             # at unpickling time (see load_build below).
00396             #
00397             # Note that no standard __newobj__ implementation exists;
00398             # you have to provide your own.  This is to enforce
00399             # compatibility with Python 2.2 (pickles written using
00400             # protocol 0 or 1 in Python 2.3 should be unpicklable by
00401             # Python 2.2).
00402             cls = args[0]
00403             if not hasattr(cls, "__new__"):
00404                 raise PicklingError(
00405                     "args[0] from __newobj__ args has no __new__")
00406             if obj is not None and cls is not obj.__class__:
00407                 raise PicklingError(
00408                     "args[0] from __newobj__ args has the wrong class")
00409             args = args[1:]
00410             save(cls)
00411             save(args)
00412             write(NEWOBJ)
00413         else:
00414             save(func)
00415             save(args)
00416             write(REDUCE)
00417 
00418         if obj is not None:
00419             self.memoize(obj)
00420 
00421         # More new special cases (that work with older protocols as
00422         # well): when __reduce__ returns a tuple with 4 or 5 items,
00423         # the 4th and 5th item should be iterators that provide list
00424         # items and dict items (as (key, value) tuples), or None.
00425 
00426         if listitems is not None:
00427             self._batch_appends(listitems)
00428 
00429         if dictitems is not None:
00430             self._batch_setitems(dictitems)
00431 
00432         if state is not None:
00433             save(state)
00434             write(BUILD)

Here is the call graph for this function:

Here is the caller graph for this function:

def pickle._Pickler.save_str (   self,
  obj,
  pack = struct.pack 
)

Definition at line 500 of file pickle.py.

00500 
00501     def save_str(self, obj, pack=struct.pack):
00502         if self.bin:
00503             encoded = obj.encode('utf-8', 'surrogatepass')
00504             n = len(encoded)
00505             self.write(BINUNICODE + pack("<i", n) + encoded)
00506         else:
00507             obj = obj.replace("\\", "\\u005c")
00508             obj = obj.replace("\n", "\\u000a")
00509             self.write(UNICODE + bytes(obj.encode('raw-unicode-escape')) +
00510                        b'\n')
        self.memoize(obj)

Here is the call graph for this function:

def pickle._Pickler.save_tuple (   self,
  obj 
)

Definition at line 513 of file pickle.py.

00513 
00514     def save_tuple(self, obj):
00515         write = self.write
00516         proto = self.proto
00517 
00518         n = len(obj)
00519         if n == 0:
00520             if proto:
00521                 write(EMPTY_TUPLE)
00522             else:
00523                 write(MARK + TUPLE)
00524             return
00525 
00526         save = self.save
00527         memo = self.memo
00528         if n <= 3 and proto >= 2:
00529             for element in obj:
00530                 save(element)
00531             # Subtle.  Same as in the big comment below.
00532             if id(obj) in memo:
00533                 get = self.get(memo[id(obj)][0])
00534                 write(POP * n + get)
00535             else:
00536                 write(_tuplesize2code[n])
00537                 self.memoize(obj)
00538             return
00539 
00540         # proto 0 or proto 1 and tuple isn't empty, or proto > 1 and tuple
00541         # has more than 3 elements.
00542         write(MARK)
00543         for element in obj:
00544             save(element)
00545 
00546         if id(obj) in memo:
00547             # Subtle.  d was not in memo when we entered save_tuple(), so
00548             # the process of saving the tuple's elements must have saved
00549             # the tuple itself:  the tuple is recursive.  The proper action
00550             # now is to throw away everything we put on the stack, and
00551             # simply GET the tuple (it's already constructed).  This check
00552             # could have been done in the "for element" loop instead, but
00553             # recursive tuples are a rare thing.
00554             get = self.get(memo[id(obj)][0])
00555             if proto:
00556                 write(POP_MARK + get)
00557             else:   # proto 0 -- POP_MARK not available
00558                 write(POP * (n+1) + get)
00559             return
00560 
00561         # No recursion.
00562         self.write(TUPLE)
00563         self.memoize(obj)

Here is the call graph for this function:


Member Data Documentation

int pickle._Pickler._BATCHSIZE = 1000 [static, private]

Definition at line 579 of file pickle.py.

Definition at line 213 of file pickle.py.

Definition at line 437 of file pickle.py.

Definition at line 214 of file pickle.py.

Definition at line 215 of file pickle.py.

Definition at line 211 of file pickle.py.

Definition at line 212 of file pickle.py.

Definition at line 208 of file pickle.py.


The documentation for this class was generated from the following file: