Back to index

obnam  1.1
Public Member Functions | Public Attributes | Static Public Attributes | Private Member Functions | Private Attributes
obnamlib.repo.Repository Class Reference

List of all members.

Public Member Functions

def __init__
def setup_hooks
def checksum
def new_checksummer
def acceptable_version
def client_dir
def list_clients
def require_root_lock
def require_shared_lock
def require_client_lock
def require_open_client
def require_started_generation
def require_no_root_lock
def require_no_shared_lock
def require_no_client_lock
def lock_root
def unlock_root
def commit_root
def get_format_version
def check_format_version
def add_client
def remove_client
def shared_dirs
def lock_shared
def commit_shared
def unlock_shared
def lock_client
def unlock_client
def commit_client
def open_client
def list_generations
def get_is_checkpoint
def start_generation
def remove_generation
def get_generation_times
def listdir
def get_metadata
def create
def remove
def put_chunk_only
def put_chunk_in_shared_trees
def get_chunk
def chunk_exists
def find_chunks
def list_chunks
def remove_chunk
def get_file_chunks
def set_file_chunks
def append_file_chunks
def set_file_data
def get_file_data
def genspec
def walk

Public Attributes

 current_time
 fs
 node_size
 upload_queue_size
 lru_size
 lockmgr
 got_root_lock
 got_shared_lock
 got_client_lock
 current_client
 current_client_id
 new_generation
 added_clients
 removed_clients
 removed_generations
 client
 prev_chunkid
 chunk_idpath
 clientlist
 chunklist
 chunksums
 hooks
 format_version
 added_generations

Static Public Attributes

int format_version = 6

Private Member Functions

def _open_client_list
def _open_shared
def _write_format_version
def _really_remove_generations
def _chunk_filename

Private Attributes

 _chunks_exists

Detailed Description

Repository for backup data.

Backup data is put on a virtual file system
(obnamlib.VirtualFileSystem instance), in some form that
the API of this class does not care about.

The repository may contain data for several clients that share 
encryption keys. Each client is identified by a name.

The repository has a "root" object, which is conceptually a list of
client names.

Each client in turn is conceptually a list of generations,
which correspond to snapshots of the user data that existed
when the generation was created.

Read-only access to the repository does not require locking.
Write access may affect only the root object, or only a client's
own data, and thus locking may affect only the root, or only
the client.

When a new generation is started, it is a copy-on-write clone
of the previous generation, and the caller needs to modify
the new generation to match the current state of user data.

The file 'metadata/format' at the root of the repository contains the
version of the repository format it uses. The version is
specified using a single integer.

Definition at line 89 of file repo.py.


Constructor & Destructor Documentation

def obnamlib.repo.Repository.__init__ (   self,
  fs,
  node_size,
  upload_queue_size,
  lru_size,
  hooks,
  idpath_depth,
  idpath_bits,
  idpath_skip,
  current_time,
  lock_timeout,
  client_name 
)

Definition at line 126 of file repo.py.

00126 
00127                  lock_timeout, client_name):
00128 
00129         self.current_time = current_time
00130         self.setup_hooks(hooks or obnamlib.HookManager())
00131         self.fs = HookedFS(self, fs, self.hooks)
00132         self.node_size = node_size
00133         self.upload_queue_size = upload_queue_size
00134         self.lru_size = lru_size
00135         
00136         hider = hashlib.md5()
00137         hider.update(client_name)
00138 
00139         self.lockmgr = obnamlib.LockManager(self.fs, lock_timeout, 
00140                                             hider.hexdigest())
00141 
00142         self.got_root_lock = False
00143         self._open_client_list()
00144         self.got_shared_lock = False
00145         self.got_client_lock = False
00146         self.current_client = None
00147         self.current_client_id = None
00148         self.new_generation = None
00149         self.added_clients = []
00150         self.removed_clients = []
00151         self.removed_generations = []
00152         self.client = None
00153         self._open_shared()
00154         self.prev_chunkid = None
00155         self.chunk_idpath = larch.IdPath('chunks', idpath_depth, 
00156                                          idpath_bits, idpath_skip)
00157         self._chunks_exists = False


Member Function Documentation

def obnamlib.repo.Repository._chunk_filename (   self,
  chunkid 
) [private]

Definition at line 618 of file repo.py.

00618 
00619     def _chunk_filename(self, chunkid):
00620         return self.chunk_idpath.convert(chunkid)

Here is the caller graph for this function:

Definition at line 158 of file repo.py.

00158 
00159     def _open_client_list(self):
00160         self.clientlist = obnamlib.ClientList(self.fs, self.node_size, 
00161                                               self.upload_queue_size, 
00162                                               self.lru_size, self)

Here is the caller graph for this function:

def obnamlib.repo.Repository._open_shared (   self) [private]

Definition at line 163 of file repo.py.

00163 
00164     def _open_shared(self):
00165         self.chunklist = obnamlib.ChunkList(self.fs, self.node_size, 
00166                                             self.upload_queue_size, 
00167                                             self.lru_size, self)
00168         self.chunksums = obnamlib.ChecksumTree(self.fs, 'chunksums', 
00169                                                len(self.checksum('')),
00170                                                self.node_size, 
00171                                                self.upload_queue_size, 
00172                                                self.lru_size, self)

Here is the caller graph for this function:

def obnamlib.repo.Repository._really_remove_generations (   self,
  remove_genids 
) [private]
Really remove a list of generations.

This is not part of the public API.

This does not make any safety checks.

Definition at line 521 of file repo.py.

00521 
00522     def _really_remove_generations(self, remove_genids):
00523         '''Really remove a list of generations.
00524         
00525         This is not part of the public API.
00526         
00527         This does not make any safety checks.
00528         
00529         '''
00530 
00531         def find_chunkids_in_gens(genids):
00532             chunkids = set()
00533             for genid in genids:
00534                 x = self.client.list_chunks_in_generation(genid)
00535                 chunkids = chunkids.union(set(x))
00536             return chunkids
00537 
00538         def find_gens_to_keep():
00539             return [genid
00540                     for genid in self.list_generations()
00541                     if genid not in remove_genids]
00542 
00543         def remove_chunks(chunk_ids):
00544             for chunk_id in chunk_ids:
00545                 try:
00546                     checksum = self.chunklist.get_checksum(chunk_id)
00547                 except KeyError:
00548                     # No checksum, therefore it can't be shared, therefore
00549                     # we can remove it.
00550                     self.remove_chunk(chunk_id)
00551                 else:
00552                     self.chunksums.remove(checksum, chunk_id, 
00553                                           self.current_client_id)
00554                     if not self.chunksums.chunk_is_used(checksum, chunk_id):
00555                         self.remove_chunk(chunk_id)
00556 
00557         def remove_gens(genids):
00558             if self.new_generation is None:
00559                 self.client.start_changes(create_tree=False)
00560             for genid in genids:
00561                 self.client.remove_generation(genid)
00562 
00563         if not remove_genids:
00564             return
00565 
00566         self.require_client_lock()
00567         self.require_shared_lock()
00568 
00569         maybe_remove = find_chunkids_in_gens(remove_genids)
00570         keep_genids = find_gens_to_keep()
00571         keep = find_chunkids_in_gens(keep_genids)
00572         remove = maybe_remove.difference(keep)
00573         remove_chunks(remove)
00574         remove_gens(remove_genids)

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository._write_format_version (   self,
  version 
) [private]
Write the desired format version to the repository.

Definition at line 324 of file repo.py.

00324 
00325     def _write_format_version(self, version):
00326         '''Write the desired format version to the repository.'''
00327         tracing.trace('write format version')
00328         if not self.fs.exists('metadata'):
00329             self.fs.mkdir('metadata')
00330         self.fs.overwrite_file('metadata/format', '%s\n' % version,
00331                                runfilters=False)

Here is the caller graph for this function:

def obnamlib.repo.Repository.acceptable_version (   self,
  version 
)
Are we compatible with on-disk format?

Definition at line 195 of file repo.py.

00195 
00196     def acceptable_version(self, version):
00197         '''Are we compatible with on-disk format?'''
00198         return self.format_version == version

Here is the caller graph for this function:

def obnamlib.repo.Repository.add_client (   self,
  client_name 
)
Add a new client to the repository.

Definition at line 346 of file repo.py.

00346 
00347     def add_client(self, client_name):
00348         '''Add a new client to the repository.'''
00349         tracing.trace('client_name=%s', client_name)
00350         self.require_root_lock()
00351         if client_name in self.list_clients():
00352             raise obnamlib.Error('client %s already exists in repository' % 
00353                                  client_name)
00354         self.added_clients.append(client_name)
        

Here is the call graph for this function:

def obnamlib.repo.Repository.append_file_chunks (   self,
  filename,
  chunkids 
)
Append to list of ids of chunks belonging to a file.

File must be in the started generation.

Definition at line 742 of file repo.py.

00742 
00743     def append_file_chunks(self, filename, chunkids):
00744         '''Append to list of ids of chunks belonging to a file.
00745         
00746         File must be in the started generation.
00747         
00748         '''
00749         
00750         self.require_started_generation()
00751         self.client.append_file_chunks(filename, chunkids)

Here is the call graph for this function:

Verify that on-disk format version is compatbile.

If not, raise BadFormat.

Definition at line 332 of file repo.py.

00332 
00333     def check_format_version(self):
00334         '''Verify that on-disk format version is compatbile.
00335         
00336         If not, raise BadFormat.
00337         
00338         '''
00339         
00340         on_disk = self.get_format_version()
00341         if on_disk is not None and not self.acceptable_version(on_disk):
00342             raise BadFormat('On-disk repository format %s is incompatible '
00343                             'with program format %s; you need to use a '
00344                             'different version of Obnam' %
00345                                 (on_disk, self.format_version))
        

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.checksum (   self,
  data 
)
Return checksum of data.

The checksum is (currently) MD5.

Definition at line 180 of file repo.py.

00180 
00181     def checksum(self, data):
00182         '''Return checksum of data.
00183         
00184         The checksum is (currently) MD5.
00185         
00186         '''
00187 
00188         checksummer = self.new_checksummer()
00189         checksummer.update(data)
00190         return checksummer.hexdigest()

Here is the call graph for this function:

def obnamlib.repo.Repository.chunk_exists (   self,
  chunkid 
)
Does a chunk exist in the repository?

Definition at line 680 of file repo.py.

00680 
00681     def chunk_exists(self, chunkid):
00682         '''Does a chunk exist in the repository?'''
00683         self.require_open_client()
00684         return self.fs.exists(self._chunk_filename(chunkid))
        

Here is the call graph for this function:

def obnamlib.repo.Repository.client_dir (   self,
  client_id 
)
Return name of sub-directory for a given client.

Definition at line 199 of file repo.py.

00199 
00200     def client_dir(self, client_id):
00201         '''Return name of sub-directory for a given client.'''
00202         return str(client_id)

Here is the caller graph for this function:

def obnamlib.repo.Repository.commit_client (   self,
  checkpoint = False 
)
Commit changes to and unlock currently locked client.

Definition at line 464 of file repo.py.

00464 
00465     def commit_client(self, checkpoint=False):
00466         '''Commit changes to and unlock currently locked client.'''
00467         tracing.trace('committing client (checkpoint=%s)', checkpoint)
00468         self.require_client_lock()
00469         self.require_shared_lock()
00470         commit_client = self.new_generation or self.removed_generations
00471         if self.new_generation:
00472             self.client.set_current_generation_is_checkpoint(checkpoint)
00473         self.added_generations = []
00474         self._really_remove_generations(self.removed_generations)
00475         if commit_client:
00476             self.client.commit()
00477         self.unlock_client()
        

Here is the call graph for this function:

Commit changes to root node, and unlock it.

Definition at line 284 of file repo.py.

00284 
00285     def commit_root(self):
00286         '''Commit changes to root node, and unlock it.'''
00287         tracing.trace('committing root')
00288         self.require_root_lock()
00289         for client_name in self.added_clients:
00290             self.clientlist.add_client(client_name)
00291             self.hooks.call('repository-add-client', 
00292                             self.clientlist, client_name)
00293         self.added_clients = []
00294         for client_name in self.removed_clients:
00295             client_id = self.clientlist.get_client_id(client_name)
00296             client_dir = self.client_dir(client_id)
00297             if client_id is not None and self.fs.exists(client_dir):
00298                 self.fs.rmtree(client_dir)
00299             self.clientlist.remove_client(client_name)
00300         self.clientlist.commit()
00301         self.unlock_root()
        

Here is the call graph for this function:

Commit changes to shared B-trees.

Definition at line 399 of file repo.py.

00399 
00400     def commit_shared(self):
00401         '''Commit changes to shared B-trees.'''
00402         
00403         tracing.trace('committing shared')
00404         self.require_shared_lock()
00405         self.chunklist.commit()
00406         self.chunksums.commit()
00407         self.unlock_shared()

Here is the call graph for this function:

def obnamlib.repo.Repository.create (   self,
  filename,
  metadata 
)
Create a new (empty) file in the new generation.

Definition at line 607 of file repo.py.

00607 
00608     def create(self, filename, metadata):
00609         '''Create a new (empty) file in the new generation.'''
00610         self.require_started_generation()
00611         encoded = obnamlib.encode_metadata(metadata)
00612         self.client.create(filename, encoded)

Here is the call graph for this function:

def obnamlib.repo.Repository.find_chunks (   self,
  checksum 
)
Return identifiers of chunks with given checksum.

Because of hash collisions, the list may be longer than one.

Definition at line 685 of file repo.py.

00685 
00686     def find_chunks(self, checksum):
00687         '''Return identifiers of chunks with given checksum.
00688         
00689         Because of hash collisions, the list may be longer than one.
00690         
00691         '''
00692 
00693         self.require_open_client()
00694         return self.chunksums.find(checksum)

Here is the call graph for this function:

def obnamlib.repo.Repository.genspec (   self,
  spec 
)
Interpret a generation specification.

Definition at line 762 of file repo.py.

00762 
00763     def genspec(self, spec):
00764         '''Interpret a generation specification.'''
00765 
00766         self.require_open_client()
00767         gens = self.list_generations()
00768         if not gens:
00769             raise obnamlib.Error('No generations')
00770         if spec == 'latest':
00771             return gens[-1]
00772         else:
00773             try:
00774                 intspec = int(spec)
00775             except ValueError:
00776                 raise obnamlib.Error('Generation %s is not an integer' % spec)
00777             if intspec in gens:
00778                 return intspec
00779             else:
00780                 raise obnamlib.Error('Generation %s not found' % spec)

Here is the call graph for this function:

def obnamlib.repo.Repository.get_chunk (   self,
  chunkid 
)
Return data of chunk with given id.

Definition at line 675 of file repo.py.

00675 
00676     def get_chunk(self, chunkid):
00677         '''Return data of chunk with given id.'''
00678         self.require_open_client()
00679         return self.fs.cat(self._chunk_filename(chunkid))
        

Here is the call graph for this function:

def obnamlib.repo.Repository.get_file_chunks (   self,
  gen,
  filename 
)
Return list of ids of chunks belonging to a file.

Definition at line 727 of file repo.py.

00727 
00728     def get_file_chunks(self, gen, filename):
00729         '''Return list of ids of chunks belonging to a file.'''
00730         self.require_open_client()
00731         return self.client.get_file_chunks(gen, filename)

Here is the call graph for this function:

def obnamlib.repo.Repository.get_file_data (   self,
  gen,
  filename 
)
Returned contents of file stored in B-tree instead of chunks dir.

Definition at line 757 of file repo.py.

00757 
00758     def get_file_data(self, gen, filename): # pragma: no cover
00759         '''Returned contents of file stored in B-tree instead of chunks dir.'''
00760         self.require_open_client()
00761         return self.client.get_file_data(gen, filename)
        

Here is the call graph for this function:

Return (major, minor) of the on-disk format version.

If on-disk repository does not have a version yet, return None.

Definition at line 302 of file repo.py.

00302 
00303     def get_format_version(self):
00304         '''Return (major, minor) of the on-disk format version.
00305         
00306         If on-disk repository does not have a version yet, return None.
00307         
00308         '''
00309         
00310         if self.fs.exists('metadata/format'):
00311             data = self.fs.cat('metadata/format', runfilters=False)
00312             lines = data.splitlines()
00313             line = lines[0]
00314             try:
00315                 version = int(line)
00316             except ValueError, e: # pragma: no cover
00317                 msg = ('Invalid repository format version (%s) -- '
00318                             'forgot encryption?' %
00319                        repr(line))
00320                 raise obnamlib.Error(msg)
00321             return version
00322         else:
00323             return None
        

Here is the caller graph for this function:

Return start and end times of a generation.

An unfinished generation has no end time, so None is returned.

Definition at line 582 of file repo.py.

00582 
00583     def get_generation_times(self, gen):
00584         '''Return start and end times of a generation.
00585         
00586         An unfinished generation has no end time, so None is returned.
00587         
00588         '''
00589 
00590         self.require_open_client()
00591         return self.client.get_generation_times(gen)

Here is the call graph for this function:

def obnamlib.repo.Repository.get_is_checkpoint (   self,
  genid 
)
Is a generation a checkpoint one?

Definition at line 499 of file repo.py.

00499 
00500     def get_is_checkpoint(self, genid):
00501         '''Is a generation a checkpoint one?'''
00502         self.require_open_client()
00503         return self.client.get_is_checkpoint(genid)
        

Here is the call graph for this function:

def obnamlib.repo.Repository.get_metadata (   self,
  gen,
  filename 
)
Return metadata for a file in a generation.

Definition at line 597 of file repo.py.

00597 
00598     def get_metadata(self, gen, filename):
00599         '''Return metadata for a file in a generation.'''
00600 
00601         self.require_open_client()
00602         try:
00603             encoded = self.client.get_metadata(gen, filename)
00604         except KeyError:
00605             raise obnamlib.Error('%s does not exist' % filename)
00606         return obnamlib.decode_metadata(encoded)

Here is the call graph for this function:

Here is the caller graph for this function:

Return list of ids of all chunks in repository.

Definition at line 695 of file repo.py.

00695 
00696     def list_chunks(self):
00697         '''Return list of ids of all chunks in repository.'''
00698         result = []
00699         pat = re.compile(r'^.*/.*/[0-9a-fA-F]+$')
00700         if self.fs.exists('chunks'):
00701             for pathname, st in self.fs.scan_tree('chunks'):
00702                 if stat.S_ISREG(st.st_mode) and pat.match(pathname):
00703                     basename = os.path.basename(pathname)
00704                     result.append(int(basename, 16))
00705         return result

Return list of names of clients using this repository.

Definition at line 203 of file repo.py.

00203 
00204     def list_clients(self):
00205         '''Return list of names of clients using this repository.'''
00206 
00207         self.check_format_version()
00208         listed = set(self.clientlist.list_clients())
00209         added = set(self.added_clients)
00210         removed = set(self.removed_clients)
00211         clients = listed.union(added).difference(removed)
00212         return list(clients)

Here is the call graph for this function:

Here is the caller graph for this function:

List existing generations for currently open client.

Definition at line 494 of file repo.py.

00494 
00495     def list_generations(self):
00496         '''List existing generations for currently open client.'''
00497         self.require_open_client()
00498         return self.client.list_generations()
        

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.listdir (   self,
  gen,
  dirname 
)
Return list of basenames in a directory within generation.

Definition at line 592 of file repo.py.

00592 
00593     def listdir(self, gen, dirname):
00594         '''Return list of basenames in a directory within generation.'''
00595         self.require_open_client()
00596         return self.client.listdir(gen, dirname)
        

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.lock_client (   self,
  client_name 
)
Lock a client for exclusive write access.

Raise obnamlib.LockFail if locking fails. Lock will be released
by commit_client() or unlock_client().

Definition at line 416 of file repo.py.

00416 
00417     def lock_client(self, client_name):
00418         '''Lock a client for exclusive write access.
00419         
00420         Raise obnamlib.LockFail if locking fails. Lock will be released
00421         by commit_client() or unlock_client().
00422 
00423         '''
00424 
00425         tracing.trace('client_name=%s', client_name)
00426         self.require_no_client_lock()
00427         self.require_no_shared_lock()
00428         
00429         self.check_format_version()
00430         client_id = self.clientlist.get_client_id(client_name)
00431         if client_id is None:
00432             raise LockFail('client %s does not exist' % client_name)
00433 
00434         client_dir = self.client_dir(client_id)
00435         if not self.fs.exists(client_dir):
00436             self.fs.mkdir(client_dir)
00437             self.hooks.call('repository-toplevel-init', self, client_dir)
00438 
00439         self.lockmgr.lock([client_dir])
00440         self.got_client_lock = True
00441         self.current_client = client_name
00442         self.current_client_id = client_id
00443         self.added_generations = []
00444         self.removed_generations = []
00445         self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, 
00446                                                   self.node_size,
00447                                                   self.upload_queue_size, 
00448                                                   self.lru_size, self)
00449         self.client.init_forest()

Here is the call graph for this function:

Lock root node.

Raise obnamlib.LockFail if locking fails. Lock will be released
by commit_root() or unlock_root().

Definition at line 253 of file repo.py.

00253 
00254     def lock_root(self):
00255         '''Lock root node.
00256         
00257         Raise obnamlib.LockFail if locking fails. Lock will be released
00258         by commit_root() or unlock_root().
00259         
00260         '''
00261 
00262         tracing.trace('locking root')
00263         self.require_no_root_lock()
00264         self.require_no_client_lock()
00265         self.require_no_shared_lock()
00266 
00267         self.lockmgr.lock(['.'])
00268         self.check_format_version()
00269         self.got_root_lock = True
00270         self.added_clients = []
00271         self.removed_clients = []
00272         self._write_format_version(self.format_version)
00273         self.clientlist.start_changes()

Here is the call graph for this function:

Lock a client for exclusive write access.

Raise obnamlib.LockFail if locking fails. Lock will be released
by commit_client() or unlock_client().

Definition at line 374 of file repo.py.

00374 
00375     def lock_shared(self):
00376         '''Lock a client for exclusive write access.
00377         
00378         Raise obnamlib.LockFail if locking fails. Lock will be released
00379         by commit_client() or unlock_client().
00380 
00381         '''
00382 
00383         tracing.trace('locking shared')
00384         self.require_no_shared_lock()
00385         self.check_format_version()
00386         self.lockmgr.lock(self.shared_dirs)
00387         self.got_shared_lock = True
00388         tracing.trace('starting changes in chunksums and chunklist')
00389         self.chunksums.start_changes()
00390         self.chunklist.start_changes()
00391         
00392         # Initialize the chunks directory for encryption, etc, if it just
00393         # got created.
00394         dirname = self.chunk_idpath.dirname
00395         filenames = self.fs.listdir(dirname)
00396         if filenames == [] or filenames == ['lock']:
00397             self.hooks.call('repository-toplevel-init', self, dirname)
00398 

Here is the call graph for this function:

Return a new checksum algorithm.

Definition at line 191 of file repo.py.

00191 
00192     def new_checksummer(self):
00193         '''Return a new checksum algorithm.'''
00194         return hashlib.md5()

Here is the caller graph for this function:

def obnamlib.repo.Repository.open_client (   self,
  client_name 
)
Open a client for read-only operation.

Definition at line 478 of file repo.py.

00478 
00479     def open_client(self, client_name):
00480         '''Open a client for read-only operation.'''
00481         tracing.trace('open r/o client_name=%s' % client_name)
00482         self.check_format_version()
00483         client_id = self.clientlist.get_client_id(client_name)
00484         if client_id is None:
00485             raise obnamlib.Error('%s is not an existing client' % client_name)
00486         self.current_client = client_name
00487         self.current_client_id = client_id
00488         client_dir = self.client_dir(client_id)
00489         self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, 
00490                                                   self.node_size, 
00491                                                   self.upload_queue_size, 
00492                                                   self.lru_size, self)
00493         self.client.init_forest()
        

Here is the call graph for this function:

def obnamlib.repo.Repository.put_chunk_in_shared_trees (   self,
  chunkid,
  checksum 
)
Put the chunk into the shared trees.

The chunk is assumed to already exist in the repository, so we
just need to add it to the shared trees that map chunkids to
checksums and checksums to chunkids.

Definition at line 657 of file repo.py.

00657 
00658     def put_chunk_in_shared_trees(self, chunkid, checksum):
00659         '''Put the chunk into the shared trees.
00660         
00661         The chunk is assumed to already exist in the repository, so we
00662         just need to add it to the shared trees that map chunkids to
00663         checksums and checksums to chunkids.
00664         
00665         '''
00666 
00667         tracing.trace('chunkid=%s', chunkid)
00668         tracing.trace('checksum=%s', repr(checksum))
00669 
00670         self.require_started_generation()
00671         self.require_shared_lock()
00672 
00673         self.chunklist.add(chunkid, checksum)
00674         self.chunksums.add(checksum, chunkid, self.current_client_id)
        

Here is the call graph for this function:

def obnamlib.repo.Repository.put_chunk_only (   self,
  data 
)
Put chunk of data into repository.

If the same data is already in the repository, it will be put there
a second time. It is the caller's responsibility to check
that the data is not already in the repository.

Return the unique identifier of the new chunk.

Definition at line 621 of file repo.py.

00621 
00622     def put_chunk_only(self, data):
00623         '''Put chunk of data into repository.
00624         
00625         If the same data is already in the repository, it will be put there
00626         a second time. It is the caller's responsibility to check
00627         that the data is not already in the repository.
00628         
00629         Return the unique identifier of the new chunk.
00630         
00631         '''
00632         
00633         def random_chunkid():
00634             return random.randint(0, obnamlib.MAX_ID)
00635         
00636         self.require_started_generation()
00637 
00638         if self.prev_chunkid is None:
00639             self.prev_chunkid = random_chunkid()
00640 
00641         while True:
00642             chunkid = (self.prev_chunkid + 1) % obnamlib.MAX_ID
00643             filename = self._chunk_filename(chunkid)
00644             try:
00645                 self.fs.write_file(filename, data)
00646             except OSError, e: # pragma: no cover
00647                 if e.errno == errno.EEXIST:
00648                     self.prev_chunkid = random_chunkid()
00649                     continue
00650                 raise
00651             else:
00652                 tracing.trace('chunkid=%s', chunkid)
00653                 break
00654 
00655         self.prev_chunkid = chunkid
00656         return chunkid

Here is the call graph for this function:

def obnamlib.repo.Repository.remove (   self,
  filename 
)
Remove file or directory or directory tree from generation.

Definition at line 613 of file repo.py.

00613 
00614     def remove(self, filename):
00615         '''Remove file or directory or directory tree from generation.'''
00616         self.require_started_generation()
00617         self.client.remove(filename)

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.remove_chunk (   self,
  chunk_id 
)
Remove a chunk from the repository.

Note that this does _not_ remove the chunk from the chunk
checksum forest. The caller is not supposed to call us until
the chunk is not there anymore.

However, it does remove the chunk from the chunk list forest.

Definition at line 706 of file repo.py.

00706 
00707     def remove_chunk(self, chunk_id):
00708         '''Remove a chunk from the repository.
00709         
00710         Note that this does _not_ remove the chunk from the chunk
00711         checksum forest. The caller is not supposed to call us until
00712         the chunk is not there anymore.
00713         
00714         However, it does remove the chunk from the chunk list forest.
00715         
00716         '''
00717 
00718         tracing.trace('chunk_id=%s', chunk_id)
00719         self.require_open_client()
00720         self.require_shared_lock()
00721         self.chunklist.remove(chunk_id)
00722         filename = self._chunk_filename(chunk_id)
00723         try:
00724             self.fs.remove(filename)
00725         except OSError:
00726             pass

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.remove_client (   self,
  client_name 
)
Remove a client from the repository.

This removes all data related to the client, including all
actual file data unless other clients also use it.

Definition at line 355 of file repo.py.

00355 
00356     def remove_client(self, client_name):
00357         '''Remove a client from the repository.
00358         
00359         This removes all data related to the client, including all
00360         actual file data unless other clients also use it.
00361         
00362         '''
00363         
00364         tracing.trace('client_name=%s', client_name)
00365         self.require_root_lock()
00366         if client_name not in self.list_clients():
00367             raise obnamlib.Error('client %s does not exist' % client_name)
00368         self.removed_clients.append(client_name)

Here is the call graph for this function:

def obnamlib.repo.Repository.remove_generation (   self,
  gen 
)
Remove a committed generation.

Definition at line 575 of file repo.py.

00575 
00576     def remove_generation(self, gen):
00577         '''Remove a committed generation.'''
00578         self.require_client_lock()
00579         if gen == self.new_generation:
00580             raise obnamlib.Error('cannot remove started generation')
00581         self.removed_generations.append(gen)

Here is the call graph for this function:

Ensure we have the lock on the currently open client.

Definition at line 223 of file repo.py.

00223 
00224     def require_client_lock(self):
00225         '''Ensure we have the lock on the currently open client.'''
00226         if not self.got_client_lock:
00227             raise LockFail('have not got lock on client')

Here is the caller graph for this function:

Ensure we haven't locked the per-client B-tree yet.

Definition at line 248 of file repo.py.

00248 
00249     def require_no_client_lock(self):
00250         '''Ensure we haven't locked the per-client B-tree yet.'''
00251         if self.got_client_lock:
00252             raise obnamlib.Error('We have already locked the client, oops')

Here is the caller graph for this function:

Ensure we haven't locked root yet.

Definition at line 238 of file repo.py.

00238 
00239     def require_no_root_lock(self):
00240         '''Ensure we haven't locked root yet.'''
00241         if self.got_root_lock:
00242             raise obnamlib.Error('We have already locked root, oops')

Here is the caller graph for this function:

Ensure we haven't locked shared B-trees yet.

Definition at line 243 of file repo.py.

00243 
00244     def require_no_shared_lock(self):
00245         '''Ensure we haven't locked shared B-trees yet.'''
00246         if self.got_shared_lock:
00247             raise obnamlib.Error('We have already locked shared B-trees, oops')

Here is the caller graph for this function:

Ensure we have opened the client (r/w or r/o).

Definition at line 228 of file repo.py.

00228 
00229     def require_open_client(self):
00230         '''Ensure we have opened the client (r/w or r/o).'''
00231         if self.current_client is None:
00232             raise obnamlib.Error('client is not open')

Here is the caller graph for this function:

Ensure we have the lock on the repository's root node.

Definition at line 213 of file repo.py.

00213 
00214     def require_root_lock(self):
00215         '''Ensure we have the lock on the repository's root node.'''
00216         if not self.got_root_lock:
00217             raise LockFail('have not got lock on root node')

Here is the caller graph for this function:

Ensure we have the lock on the shared B-trees except clientlist.

Definition at line 218 of file repo.py.

00218 
00219     def require_shared_lock(self):
00220         '''Ensure we have the lock on the shared B-trees except clientlist.'''
00221         if not self.got_shared_lock:
00222             raise LockFail('have not got lock on shared B-trees')

Here is the caller graph for this function:

Ensure we have started a new generation.

Definition at line 233 of file repo.py.

00233 
00234     def require_started_generation(self):
00235         '''Ensure we have started a new generation.'''
00236         if self.new_generation is None:
00237             raise obnamlib.Error('new generation has not started')

Here is the caller graph for this function:

def obnamlib.repo.Repository.set_file_chunks (   self,
  filename,
  chunkids 
)
Set ids of chunks belonging to a file.

File must be in the started generation.

Definition at line 732 of file repo.py.

00732 
00733     def set_file_chunks(self, filename, chunkids):
00734         '''Set ids of chunks belonging to a file.
00735         
00736         File must be in the started generation.
00737         
00738         '''
00739         
00740         self.require_started_generation()
00741         self.client.set_file_chunks(filename, chunkids)

Here is the call graph for this function:

def obnamlib.repo.Repository.set_file_data (   self,
  filename,
  contents 
)
Store contents of file in B-tree instead of chunks dir.

Definition at line 752 of file repo.py.

00752 
00753     def set_file_data(self, filename, contents): # pragma: no cover
00754         '''Store contents of file in B-tree instead of chunks dir.'''
00755         self.require_started_generation()
00756         self.client.set_file_data(filename, contents)

Here is the call graph for this function:

def obnamlib.repo.Repository.setup_hooks (   self,
  hooks 
)

Definition at line 173 of file repo.py.

00173 
00174     def setup_hooks(self, hooks):
00175         self.hooks = hooks
00176         
00177         self.hooks.new('repository-toplevel-init')
00178         self.hooks.new_filter('repository-data')
00179         self.hooks.new('repository-add-client')
        

Definition at line 370 of file repo.py.

00370 
00371     def shared_dirs(self):
00372         return [self.chunklist.dirname, self.chunksums.dirname,
00373                 self.chunk_idpath.dirname]
        

Here is the caller graph for this function:

Start a new generation.

The new generation is a copy-on-write clone of the previous
one (or empty, if first generation).

Definition at line 504 of file repo.py.

00504 
00505     def start_generation(self):
00506         '''Start a new generation.
00507         
00508         The new generation is a copy-on-write clone of the previous
00509         one (or empty, if first generation).
00510         
00511         '''
00512         tracing.trace('start new generation')
00513         self.require_client_lock()
00514         if self.new_generation is not None:
00515             raise obnamlib.Error('Cannot start two new generations')
00516         self.client.start_generation()
00517         self.new_generation = \
00518             self.client.get_generation_id(self.client.tree)
00519         self.added_generations.append(self.new_generation)
00520         return self.new_generation

Here is the call graph for this function:

Unlock currently locked client, without committing changes.

Definition at line 450 of file repo.py.

00450 
00451     def unlock_client(self):
00452         '''Unlock currently locked client, without committing changes.'''
00453         tracing.trace('unlocking client')
00454         self.require_client_lock()
00455         self.new_generation = None
00456         self._really_remove_generations(self.added_generations)
00457         self.lockmgr.unlock([self.client.dirname])
00458         self.client = None # FIXME: This should remove uncommitted data.
00459         self.added_generations = []
00460         self.removed_generations = []
00461         self.got_client_lock = False
00462         self.current_client = None
00463         self.current_client_id = None

Here is the call graph for this function:

Here is the caller graph for this function:

Unlock root node without committing changes made.

Definition at line 274 of file repo.py.

00274 
00275     def unlock_root(self):
00276         '''Unlock root node without committing changes made.'''
00277         tracing.trace('unlocking root')
00278         self.require_root_lock()
00279         self.added_clients = []
00280         self.removed_clients = []
00281         self.lockmgr.unlock(['.'])
00282         self.got_root_lock = False
00283         self._open_client_list()
        

Here is the call graph for this function:

Here is the caller graph for this function:

Unlock currently locked shared B-trees.

Definition at line 408 of file repo.py.

00408 
00409     def unlock_shared(self):
00410         '''Unlock currently locked shared B-trees.'''
00411         tracing.trace('unlocking shared')
00412         self.require_shared_lock()
00413         self.lockmgr.unlock(self.shared_dirs)
00414         self.got_shared_lock = False
00415         self._open_shared()
        

Here is the call graph for this function:

Here is the caller graph for this function:

def obnamlib.repo.Repository.walk (   self,
  gen,
  arg,
  depth_first = False 
)
Iterate over each pathname specified by argument.

This is a generator. Each return value is a tuple consisting
of a pathname and its corresponding metadata. Directories are
recursed into.

Definition at line 781 of file repo.py.

00781 
00782     def walk(self, gen, arg, depth_first=False):
00783         '''Iterate over each pathname specified by argument.
00784         
00785         This is a generator. Each return value is a tuple consisting
00786         of a pathname and its corresponding metadata. Directories are
00787         recursed into.
00788         
00789         '''
00790         
00791         arg = os.path.normpath(arg)
00792         metadata = self.get_metadata(gen, arg)
00793         if metadata.isdir():
00794             if not depth_first:
00795                 yield arg, metadata
00796             kids = self.listdir(gen, arg)
00797             kidpaths = [os.path.join(arg, kid) for kid in kids]
00798             for kidpath in kidpaths:
00799                 for x in self.walk(gen, kidpath, depth_first=depth_first):
00800                     yield x
00801             if depth_first:
00802                 yield arg, metadata
00803         else:
00804             yield arg, metadata
00805 

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Definition at line 156 of file repo.py.

Definition at line 148 of file repo.py.

Definition at line 442 of file repo.py.

Definition at line 154 of file repo.py.

Definition at line 164 of file repo.py.

Definition at line 167 of file repo.py.

Definition at line 151 of file repo.py.

Definition at line 159 of file repo.py.

Definition at line 145 of file repo.py.

Definition at line 146 of file repo.py.

Definition at line 128 of file repo.py.

Definition at line 122 of file repo.py.

Definition at line 197 of file repo.py.

Definition at line 130 of file repo.py.

Definition at line 144 of file repo.py.

Definition at line 141 of file repo.py.

Definition at line 143 of file repo.py.

Definition at line 174 of file repo.py.

Definition at line 138 of file repo.py.

Definition at line 133 of file repo.py.

Definition at line 147 of file repo.py.

Definition at line 131 of file repo.py.

Definition at line 153 of file repo.py.

Definition at line 149 of file repo.py.

Definition at line 150 of file repo.py.

Definition at line 132 of file repo.py.


The documentation for this class was generated from the following file: