Back to index

python3.2  3.2.2
Public Member Functions | Public Attributes | Private Member Functions | Static Private Member Functions | Private Attributes | Static Private Attributes
multiprocessing.heap.Heap Class Reference
Inheritance diagram for multiprocessing.heap.Heap:
Inheritance graph
[legend]
Collaboration diagram for multiprocessing.heap.Heap:
Collaboration graph
[legend]

List of all members.

Public Member Functions

def __init__
def free
def malloc

Public Attributes

_PyObject_HEAD_EXTRA Py_ssize_t ob_refcnt
struct _typeobjectob_type

Private Member Functions

def _malloc
def _free
def _absorb
def _free_pending_blocks

Static Private Member Functions

def _roundup

Private Attributes

 _lastpid
 _lock
 _size
 _lengths
 _len_to_seq
 _start_to_block
 _stop_to_block
 _allocated_blocks
 _arenas
 _pending_free_blocks

Static Private Attributes

int _alignment = 8

Detailed Description

Definition at line 90 of file heap.py.


Constructor & Destructor Documentation

def multiprocessing.heap.Heap.__init__ (   self,
  size = mmap.PAGESIZE 
)

Definition at line 94 of file heap.py.

00094 
00095     def __init__(self, size=mmap.PAGESIZE):
00096         self._lastpid = os.getpid()
00097         self._lock = threading.Lock()
00098         self._size = size
00099         self._lengths = []
00100         self._len_to_seq = {}
00101         self._start_to_block = {}
00102         self._stop_to_block = {}
00103         self._allocated_blocks = set()
00104         self._arenas = []
00105         # list of pending blocks to free - see free() comment below
00106         self._pending_free_blocks = []

Here is the caller graph for this function:


Member Function Documentation

def multiprocessing.heap.Heap._absorb (   self,
  block 
) [private]

Definition at line 165 of file heap.py.

00165 
00166     def _absorb(self, block):
00167         # deregister this block so it can be merged with a neighbour
00168         (arena, start, stop) = block
00169         del self._start_to_block[(arena, start)]
00170         del self._stop_to_block[(arena, stop)]
00171 
00172         length = stop - start
00173         seq = self._len_to_seq[length]
00174         seq.remove(block)
00175         if not seq:
00176             del self._len_to_seq[length]
00177             self._lengths.remove(length)
00178 
00179         return start, stop

Here is the caller graph for this function:

def multiprocessing.heap.Heap._free (   self,
  block 
) [private]

Definition at line 135 of file heap.py.

00135 
00136     def _free(self, block):
00137         # free location and try to merge with neighbours
00138         (arena, start, stop) = block
00139 
00140         try:
00141             prev_block = self._stop_to_block[(arena, start)]
00142         except KeyError:
00143             pass
00144         else:
00145             start, _ = self._absorb(prev_block)
00146 
00147         try:
00148             next_block = self._start_to_block[(arena, stop)]
00149         except KeyError:
00150             pass
00151         else:
00152             _, stop = self._absorb(next_block)
00153 
00154         block = (arena, start, stop)
00155         length = stop - start
00156 
00157         try:
00158             self._len_to_seq[length].append(block)
00159         except KeyError:
00160             self._len_to_seq[length] = [block]
00161             bisect.insort(self._lengths, length)
00162 
00163         self._start_to_block[(arena, start)] = block
00164         self._stop_to_block[(arena, stop)] = block

Here is the call graph for this function:

Here is the caller graph for this function:

Definition at line 180 of file heap.py.

00180 
00181     def _free_pending_blocks(self):
00182         # Free all the blocks in the pending list - called with the lock held.
00183         while True:
00184             try:
00185                 block = self._pending_free_blocks.pop()
00186             except IndexError:
00187                 break
00188             self._allocated_blocks.remove(block)
00189             self._free(block)

Here is the call graph for this function:

Here is the caller graph for this function:

def multiprocessing.heap.Heap._malloc (   self,
  size 
) [private]

Definition at line 113 of file heap.py.

00113 
00114     def _malloc(self, size):
00115         # returns a large enough block -- it might be much larger
00116         i = bisect.bisect_left(self._lengths, size)
00117         if i == len(self._lengths):
00118             length = self._roundup(max(self._size, size), mmap.PAGESIZE)
00119             self._size *= 2
00120             info('allocating a new mmap of length %d', length)
00121             arena = Arena(length)
00122             self._arenas.append(arena)
00123             return (arena, 0, length)
00124         else:
00125             length = self._lengths[i]
00126             seq = self._len_to_seq[length]
00127             block = seq.pop()
00128             if not seq:
00129                 del self._len_to_seq[length], self._lengths[i]
00130 
00131         (arena, start, stop) = block
00132         del self._start_to_block[(arena, start)]
00133         del self._stop_to_block[(arena, stop)]
00134         return block

Here is the call graph for this function:

Here is the caller graph for this function:

def multiprocessing.heap.Heap._roundup (   n,
  alignment 
) [static, private]

Definition at line 108 of file heap.py.

00108 
00109     def _roundup(n, alignment):
00110         # alignment must be a power of 2
00111         mask = alignment - 1
00112         return (n + mask) & ~mask

Here is the caller graph for this function:

def multiprocessing.heap.Heap.free (   self,
  block 
)

Definition at line 190 of file heap.py.

00190 
00191     def free(self, block):
00192         # free a block returned by malloc()
00193         # Since free() can be called asynchronously by the GC, it could happen
00194         # that it's called while self._lock is held: in that case,
00195         # self._lock.acquire() would deadlock (issue #12352). To avoid that, a
00196         # trylock is used instead, and if the lock can't be acquired
00197         # immediately, the block is added to a list of blocks to be freed
00198         # synchronously sometimes later from malloc() or free(), by calling
00199         # _free_pending_blocks() (appending and retrieving from a list is not
00200         # strictly thread-safe but under cPython it's atomic thanks to the GIL).
00201         assert os.getpid() == self._lastpid
00202         if not self._lock.acquire(False):
00203             # can't acquire the lock right now, add the block to the list of
00204             # pending blocks to free
00205             self._pending_free_blocks.append(block)
00206         else:
00207             # we hold the lock
00208             try:
00209                 self._free_pending_blocks()
00210                 self._allocated_blocks.remove(block)
00211                 self._free(block)
00212             finally:
00213                 self._lock.release()

Here is the call graph for this function:

def multiprocessing.heap.Heap.malloc (   self,
  size 
)

Definition at line 214 of file heap.py.

00214 
00215     def malloc(self, size):
00216         # return a block of right size (possibly rounded up)
00217         assert 0 <= size < sys.maxsize
00218         if os.getpid() != self._lastpid:
00219             self.__init__()                     # reinitialize after fork
00220         self._lock.acquire()
00221         self._free_pending_blocks()
00222         try:
00223             size = self._roundup(max(size,1), self._alignment)
00224             (arena, start, stop) = self._malloc(size)
00225             new_stop = start + size
00226             if new_stop < stop:
00227                 self._free((arena, new_stop, stop))
00228             block = (arena, start, new_stop)
00229             self._allocated_blocks.add(block)
00230             return block
00231         finally:
00232             self._lock.release()
00233 
00234 #
00235 # Class representing a chunk of an mmap -- can be inherited
00236 #


Member Data Documentation

Definition at line 92 of file heap.py.

Definition at line 102 of file heap.py.

Definition at line 103 of file heap.py.

Definition at line 95 of file heap.py.

Definition at line 99 of file heap.py.

Definition at line 98 of file heap.py.

Definition at line 96 of file heap.py.

Definition at line 105 of file heap.py.

Definition at line 97 of file heap.py.

Definition at line 100 of file heap.py.

Definition at line 101 of file heap.py.

Definition at line 107 of file object.h.

struct _typeobject* _object::ob_type [inherited]

Definition at line 108 of file object.h.


The documentation for this class was generated from the following file: