Back to index

nordugrid-arc-nox  1.1.0~rc6
Public Member Functions | Static Public Member Functions | Protected Attributes | Private Member Functions | Static Private Member Functions | Private Attributes | Static Private Attributes
Arc::DataPointFile Class Reference

#include <DataPointFile.h>

Inheritance diagram for Arc::DataPointFile:
Inheritance graph
[legend]
Collaboration diagram for Arc::DataPointFile:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 DataPointFile (const URL &url, const UserConfig &usercfg)
virtual ~DataPointFile ()
virtual DataStatus StartReading (DataBuffer &buffer)
 Start reading data from URL.
virtual DataStatus StartWriting (DataBuffer &buffer, DataCallback *space_cb=NULL)
 Start writing data to URL.
virtual DataStatus StopReading ()
 Stop reading.
virtual DataStatus StopWriting ()
 Stop writing.
virtual DataStatus Check ()
 Query the DataPoint to check if object is accessible.
virtual DataStatus Remove ()
 Remove/delete object at URL.
virtual DataStatus ListFiles (std::list< FileInfo > &files, bool long_list=false, bool resolve=false, bool metadata=false)
 List file(s).
virtual bool WriteOutOfOrder ()
 Returns true if URL can accept scattered data for writing operation.
virtual bool IsIndex () const
 Check if URL is an Indexing Service.
virtual long long int BufSize () const
 Get suggested buffer size for transfers.
virtual int BufNum () const
 Get suggested number of buffers for transfers.
virtual bool Local () const
 Returns true if file is local, e.g. file:// urls.
virtual bool ReadOnly () const
virtual void ReadOutOfOrder (bool v)
 Allow/disallow DataPoint to produce scattered data during reading operation.
virtual void SetAdditionalChecks (bool v)
 Allow/disallow additional checks.
virtual bool GetAdditionalChecks () const
 Check if additional checks before will be performed.
virtual void SetSecure (bool v)
 Allow/disallow heavy security during data transfer.
virtual bool GetSecure () const
 Check if heavy security during data transfer is allowed.
virtual void Passive (bool v)
 Request passive transfers for FTP-like protocols.
virtual void Range (unsigned long long int start=0, unsigned long long int end=0)
 Set range of bytes to retrieve.
virtual DataStatus Resolve (bool source)
 Resolves index service URL into list of ordinary URLs.
virtual bool Registered () const
 Check if file is registered in Indexing Service.
virtual DataStatus PreRegister (bool replication, bool force=false)
 Index service preregistration.
virtual DataStatus PostRegister (bool replication)
 Index Service postregistration.
virtual DataStatus PreUnregister (bool replication)
 Index Service preunregistration.
virtual DataStatus Unregister (bool all)
 Index Service unregistration.
virtual bool AcceptsMeta ()
 If endpoint can have any use from meta information.
virtual bool ProvidesMeta ()
 If endpoint can provide at least some meta information directly.
virtual const URLCurrentLocation () const
 Returns current (resolved) URL.
virtual const std::string & CurrentLocationMetadata () const
 Returns meta information used to create current URL.
virtual DataStatus CompareLocationMetadata () const
 Compare metadata of DataPoint and current location.
virtual bool NextLocation ()
 Switch to next location in list of URLs.
virtual bool LocationValid () const
 Returns false if out of retries.
virtual bool HaveLocations () const
 Returns true if number of resolved URLs is not 0.
virtual DataStatus AddLocation (const URL &url, const std::string &meta)
 Add URL to list.
virtual DataStatus RemoveLocation ()
 Remove current URL from list.
virtual DataStatus RemoveLocations (const DataPoint &p)
 Remove locations present in another DataPoint object.
virtual const URLGetURL () const
 Returns the URL that was passed to the constructor.
virtual const UserConfigGetUserConfig () const
 Returns the UserConfig that was passed to the constructor.
virtual std::string str () const
 Returns a string representation of the DataPoint.
virtual operator bool () const
 Is DataPoint valid?
virtual bool operator! () const
 Is DataPoint valid?
virtual DataStatus GetFailureReason (void) const
 Returns reason of transfer failure, as reported by callbacks. This could be different from the failure returned by the methods themselves.
virtual bool CheckSize () const
 Check if meta-information 'size' is available.
virtual void SetSize (const unsigned long long int val)
 Set value of meta-information 'size'.
virtual unsigned long long int GetSize () const
 Get value of meta-information 'size'.
virtual bool CheckCheckSum () const
 Check if meta-information 'checksum' is available.
virtual void SetCheckSum (const std::string &val)
 Set value of meta-information 'checksum'.
virtual const std::string & GetCheckSum () const
 Get value of meta-information 'checksum'.
virtual const std::string DefaultCheckSum () const
 Default checksum type.
virtual bool CheckCreated () const
 Check if meta-information 'creation/modification time' is available.
virtual void SetCreated (const Time &val)
 Set value of meta-information 'creation/modification time'.
virtual const TimeGetCreated () const
 Get value of meta-information 'creation/modification time'.
virtual bool CheckValid () const
 Check if meta-information 'validity time' is available.
virtual void SetValid (const Time &val)
 Set value of meta-information 'validity time'.
virtual const TimeGetValid () const
 Get value of meta-information 'validity time'.
virtual bool Cache () const
 Returns true if file is cacheable.
virtual int GetTries () const
 Returns number of retries left.
virtual void SetTries (const int n)
 Set number of retries.
virtual void NextTry (void)
 Decrease number of retries left.
virtual void SetMeta (const DataPoint &p)
 Copy meta information from another object.
virtual bool CompareMeta (const DataPoint &p) const
 Compare meta information from another object.

Static Public Member Functions

static PluginInstance (PluginArgument *arg)

Protected Attributes

DataBufferbuffer
long long int bufsize
int bufnum
bool local
bool readonly
bool linkable
bool is_secure
bool force_secure
bool force_passive
bool additional_checks
bool allow_out_of_order
unsigned long long int range_start
unsigned long long int range_end
const URLurl
const UserConfigusercfg
unsigned long long int size
std::string checksum
Time created
Time valid
int triesleft
DataStatus failure_code
bool cache
std::list< std::string > valid_url_options
 Subclasses should add their own specific options to this list.

Private Member Functions

void read_file ()
void write_file ()

Static Private Member Functions

static void read_file_start (void *arg)
static void write_file_start (void *arg)

Private Attributes

SimpleCondition transfer_cond
bool reading
bool writing
int fd
bool is_channel

Static Private Attributes

static Logger logger

Detailed Description

Definition at line 13 of file DataPointFile.h.


Constructor & Destructor Documentation

Arc::DataPointFile::DataPointFile ( const URL url,
const UserConfig usercfg 
)

Definition at line 37 of file DataPointFile.cpp.

    : DataPointDirect(url, usercfg),
      reading(false),
      writing(false),
      is_channel(false) {
    if (url.Protocol() == "file") {
      cache = false;
      is_channel = false;
      local = true;
    }
    else if (url.Path() == "-") { // won't work
      linkable = false;
      is_channel = true;
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Definition at line 53 of file DataPointFile.cpp.

Here is the call graph for this function:


Member Function Documentation

bool Arc::DataPointDirect::AcceptsMeta ( ) [virtual, inherited]

If endpoint can have any use from meta information.

Implements Arc::DataPoint.

Definition at line 122 of file DataPointDirect.cpp.

                                    {
    return false;
  }
DataStatus Arc::DataPointDirect::AddLocation ( const URL url,
const std::string &  meta 
) [virtual, inherited]

Add URL to list.

Parameters:
urlLocation URL to add.
metaLocation meta information.

Implements Arc::DataPoint.

Definition at line 161 of file DataPointDirect.cpp.

int Arc::DataPointDirect::BufNum ( ) const [virtual, inherited]

Get suggested number of buffers for transfers.

Implements Arc::DataPoint.

Definition at line 56 of file DataPointDirect.cpp.

                                    {
    return bufnum;
  }
long long int Arc::DataPointDirect::BufSize ( ) const [virtual, inherited]

Get suggested buffer size for transfers.

Implements Arc::DataPoint.

Definition at line 52 of file DataPointDirect.cpp.

                                               {
    return bufsize;
  }
bool Arc::DataPoint::Cache ( ) const [virtual, inherited]

Returns true if file is cacheable.

Definition at line 86 of file DataPoint.cpp.

                              {
    return cache;
  }

Here is the caller graph for this function:

Query the DataPoint to check if object is accessible.

If possible this method will also try to provide meta information about the object.

Implements Arc::DataPoint.

Definition at line 188 of file DataPointFile.cpp.

                                  {
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    User user;
    int res = user.check_file_access(url.Path(), O_RDONLY);
    if (res != 0) {
      logger.msg(INFO, "File is not accessible: %s", url.Path());
      return DataStatus::CheckError;
    }
    struct stat st;
    if (stat((url.Path()).c_str(), &st) != 0) {
      logger.msg(INFO, "Can't stat file: %s", url.Path());
      return DataStatus::CheckError;
    }
    SetSize(st.st_size);
    SetCreated(st.st_mtime);
    return DataStatus::Success;
  }

Here is the call graph for this function:

bool Arc::DataPoint::CheckCheckSum ( ) const [virtual, inherited]

Check if meta-information 'checksum' is available.

Definition at line 102 of file DataPoint.cpp.

                                      {
    return (!checksum.empty());
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckCreated ( ) const [virtual, inherited]

Check if meta-information 'creation/modification time' is available.

Definition at line 118 of file DataPoint.cpp.

                                     {
    return (created != -1);
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckSize ( ) const [virtual, inherited]

Check if meta-information 'size' is available.

Definition at line 90 of file DataPoint.cpp.

                                  {
    return (size != (unsigned long long int)(-1));
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckValid ( ) const [virtual, inherited]

Check if meta-information 'validity time' is available.

Definition at line 130 of file DataPoint.cpp.

                                   {
    return (valid != -1);
  }

Here is the caller graph for this function:

Compare metadata of DataPoint and current location.

Returns inconsistency error or error encountered during operation, or success

Implements Arc::DataPoint.

Definition at line 143 of file DataPointDirect.cpp.

bool Arc::DataPoint::CompareMeta ( const DataPoint p) const [virtual, inherited]

Compare meta information from another object.

Undefined values are not used for comparison.

Parameters:
pobject to which to compare.

Definition at line 165 of file DataPoint.cpp.

                                                      {
    if (CheckSize() && p.CheckSize())
      if (GetSize() != p.GetSize())
        return false;
    if (CheckCheckSum() && p.CheckCheckSum())
      // TODO: compare checksums properly
      if (strcasecmp(GetCheckSum().c_str(), p.GetCheckSum().c_str()))
        return false;
    if (CheckValid() && p.CheckValid())
      if (GetValid() != p.GetValid())
        return false;
    return true;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

const URL & Arc::DataPointDirect::CurrentLocation ( ) const [virtual, inherited]

Returns current (resolved) URL.

Implements Arc::DataPoint.

Definition at line 134 of file DataPointDirect.cpp.

                                                    {
    return url;
  }

Here is the caller graph for this function:

const std::string & Arc::DataPointDirect::CurrentLocationMetadata ( ) const [virtual, inherited]

Returns meta information used to create current URL.

Usage differs between different indexing services.

Implements Arc::DataPoint.

Definition at line 138 of file DataPointDirect.cpp.

                                                                  {
    static const std::string empty;
    return empty;
  }
const std::string Arc::DataPoint::DefaultCheckSum ( ) const [virtual, inherited]

Default checksum type.

Reimplemented in Arc::DataPointSRM, and Arc::DataPointLFC.

Definition at line 114 of file DataPoint.cpp.

                                                   {
    return std::string("cksum");
  }

Here is the call graph for this function:

Here is the caller graph for this function:

bool Arc::DataPointDirect::GetAdditionalChecks ( ) const [virtual, inherited]

Check if additional checks before will be performed.

Implements Arc::DataPoint.

Definition at line 80 of file DataPointDirect.cpp.

                                                  {
    return additional_checks;
  }
const std::string & Arc::DataPoint::GetCheckSum ( ) const [virtual, inherited]

Get value of meta-information 'checksum'.

Definition at line 110 of file DataPoint.cpp.

                                                {
    return checksum;
  }

Here is the caller graph for this function:

const Time & Arc::DataPoint::GetCreated ( ) const [virtual, inherited]

Get value of meta-information 'creation/modification time'.

Definition at line 126 of file DataPoint.cpp.

                                          {
    return created;
  }

Here is the caller graph for this function:

DataStatus Arc::DataPoint::GetFailureReason ( void  ) const [virtual, inherited]

Returns reason of transfer failure, as reported by callbacks. This could be different from the failure returned by the methods themselves.

Definition at line 82 of file DataPoint.cpp.

                                               {
    return failure_code;
  }

Here is the caller graph for this function:

bool Arc::DataPointDirect::GetSecure ( ) const [virtual, inherited]

Check if heavy security during data transfer is allowed.

Implements Arc::DataPoint.

Definition at line 88 of file DataPointDirect.cpp.

                                        {
    return is_secure;
  }
unsigned long long int Arc::DataPoint::GetSize ( ) const [virtual, inherited]

Get value of meta-information 'size'.

Definition at line 98 of file DataPoint.cpp.

                                                  {
    return size;
  }

Here is the caller graph for this function:

int Arc::DataPoint::GetTries ( ) const [virtual, inherited]

Returns number of retries left.

Definition at line 142 of file DataPoint.cpp.

                                {
    return triesleft;
  }

Here is the caller graph for this function:

const URL & Arc::DataPoint::GetURL ( ) const [virtual, inherited]

Returns the URL that was passed to the constructor.

Definition at line 45 of file DataPoint.cpp.

                                     {
    return url;
  }

Here is the caller graph for this function:

const UserConfig & Arc::DataPoint::GetUserConfig ( ) const [virtual, inherited]

Returns the UserConfig that was passed to the constructor.

Definition at line 49 of file DataPoint.cpp.

                                                   {
    return usercfg;
  }

Here is the caller graph for this function:

const Time & Arc::DataPoint::GetValid ( ) const [virtual, inherited]

Get value of meta-information 'validity time'.

Definition at line 138 of file DataPoint.cpp.

                                        {
    return valid;
  }

Here is the caller graph for this function:

bool Arc::DataPointDirect::HaveLocations ( ) const [virtual, inherited]

Returns true if number of resolved URLs is not 0.

Implements Arc::DataPoint.

Definition at line 157 of file DataPointDirect.cpp.

                                            {
    return true;
  }

Definition at line 58 of file DataPointFile.cpp.

                                                     {
    DataPointPluginArgument *dmcarg = dynamic_cast<DataPointPluginArgument*>(arg);
    if (!dmcarg)
      return NULL;
    if (((const URL &)(*dmcarg)).Protocol() != "file")
      return NULL;
    return new DataPointFile(*dmcarg, *dmcarg);
  }

Here is the call graph for this function:

bool Arc::DataPointDirect::IsIndex ( ) const [virtual, inherited]

Check if URL is an Indexing Service.

Implements Arc::DataPoint.

Definition at line 48 of file DataPointDirect.cpp.

                                      {
    return false;
  }
DataStatus Arc::DataPointFile::ListFiles ( std::list< FileInfo > &  files,
bool  long_list = false,
bool  resolve = false,
bool  metadata = false 
) [virtual]

List file(s).

If the DataPoint represents a directory its contents will be listed.

Parameters:
fileswill contain list of file names and optionally their attributes.
long)listif true, list additional properties of each file.
resolveif true, resolve physical locations (relevant for indexing services only).

Implements Arc::DataPoint.

Definition at line 442 of file DataPointFile.cpp.

                                                     {
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    std::string dirname = url.Path();
    if (dirname[dirname.length() - 1] == '/')
      dirname.resize(dirname.length() - 1);

    struct stat st;
    if (stat(dirname.c_str(), &st) != 0) {
      logger.msg(INFO, "Failed to read object %s: %s", dirname, strerror(errno));
      return DataStatus::ListError;
    }
    if (S_ISDIR(st.st_mode) && !metadata) {
      try {
        Glib::Dir dir(dirname);
        std::string file_name;
        while ((file_name = dir.read_name()) != "") {
          std::list<FileInfo>::iterator f =
            files.insert(files.end(), FileInfo(file_name.c_str()));
          if (long_list) {
            std::string fname = dirname + "/" + file_name;
            struct stat st;
            if (stat(fname.c_str(), &st) == 0) {
              f->SetSize(st.st_size);
              f->SetCreated(st.st_mtime);
              if (S_ISDIR(st.st_mode))
                f->SetType(FileInfo::file_type_dir);
              else if (S_ISREG(st.st_mode))
                f->SetType(FileInfo::file_type_file);
            }
          }
        }
      }
      catch (Glib::FileError& e) {
        logger.msg(INFO, "Failed to read object %s", dirname);
        return DataStatus::ListError;
      }
    }
    else {
      std::list<FileInfo>::iterator f =
        files.insert(files.end(), FileInfo(dirname));
      f->SetMetaData("path", dirname);
      f->SetSize(st.st_size);
      f->SetMetaData("size", tostring(st.st_size));
      logger.msg(INFO, "size is %s", tostring(st.st_size));
      f->SetCreated(st.st_mtime);
      f->SetMetaData("mtime", (Time(st.st_mtime)).str());
       if (S_ISDIR(st.st_mode)) {
         f->SetType(FileInfo::file_type_dir);
        f->SetMetaData("type", "dir");
      }
      else if (S_ISREG(st.st_mode)) {
        f->SetType(FileInfo::file_type_file);
        f->SetMetaData("type", "file");
      }
      // fill some more metadata
      f->SetMetaData("atime", (Time(st.st_atime)).str());
      f->SetMetaData("ctime", (Time(st.st_ctime)).str());
      f->SetMetaData("group", tostring(st.st_gid));
      f->SetMetaData("owner", tostring(st.st_uid));
      unsigned int mode = st.st_mode;
      std::string perms;
      if (mode & S_IRUSR) perms += 'r'; else perms += '-';
      if (mode & S_IWUSR) perms += 'w'; else perms += '-';
      if (mode & S_IXUSR) perms += 'x'; else perms += '-';
#ifndef WIN32
      if (mode & S_IRGRP) perms += 'r'; else perms += '-';
      if (mode & S_IWGRP) perms += 'w'; else perms += '-';
      if (mode & S_IXGRP) perms += 'x'; else perms += '-';
      if (mode & S_IROTH) perms += 'r'; else perms += '-';
      if (mode & S_IWOTH) perms += 'w'; else perms += '-';
      if (mode & S_IXOTH) perms += 'x'; else perms += '-';
#endif
      f->SetMetaData("accessperm", perms);
    }
    return DataStatus::Success;
  }

Here is the call graph for this function:

bool Arc::DataPointDirect::Local ( ) const [virtual, inherited]

Returns true if file is local, e.g. file:// urls.

Implements Arc::DataPoint.

Definition at line 60 of file DataPointDirect.cpp.

                                    {
    return local;
  }
bool Arc::DataPointDirect::LocationValid ( ) const [virtual, inherited]

Returns false if out of retries.

Implements Arc::DataPoint.

Definition at line 153 of file DataPointDirect.cpp.

                                            {
    return (triesleft > 0);
  }
bool Arc::DataPointDirect::NextLocation ( ) [virtual, inherited]

Switch to next location in list of URLs.

At last location switch to first if number of allowed retries is not exceeded. Returns false if no retries left.

Implements Arc::DataPoint.

Definition at line 147 of file DataPointDirect.cpp.

                                     {
    if (triesleft > 0)
      --triesleft;
    return (triesleft > 0);
  }
void Arc::DataPoint::NextTry ( void  ) [virtual, inherited]

Decrease number of retries left.

Definition at line 150 of file DataPoint.cpp.

                          {
    if(triesleft) --triesleft;
  }

Here is the caller graph for this function:

Arc::DataPoint::operator bool ( void  ) const [virtual, inherited]

Is DataPoint valid?

Definition at line 57 of file DataPoint.cpp.

                                 {

    if (!url)
      return false;
      
    // URL option validation. Subclasses which do not want to validate
    // URL options should override this method.
    std::map<std::string, std::string> options = url.Options();
    for (std::map<std::string, std::string>::iterator i = options.begin(); i != options.end(); i++) {
      bool valid = false; 
      for (std::list<std::string>::const_iterator j = valid_url_options.begin(); j != valid_url_options.end(); j++) {
        if (i->first == *j) valid = true;
      }
      if (!valid) {
        logger.msg(ERROR, "Invalid URL option: %s", i->first);
        return false;
      }
    }
    return true;
  }

Here is the call graph for this function:

bool Arc::DataPoint::operator! ( void  ) const [virtual, inherited]

Is DataPoint valid?

Definition at line 78 of file DataPoint.cpp.

                                  {
    return !((bool)*this);
  }
void Arc::DataPointDirect::Passive ( bool  v) [virtual, inherited]

Request passive transfers for FTP-like protocols.

Parameters:
trueto request.

Implements Arc::DataPoint.

Definition at line 92 of file DataPointDirect.cpp.

                                        {
    force_passive = val;
  }
DataStatus Arc::DataPointDirect::PostRegister ( bool  replication) [virtual, inherited]

Index Service postregistration.

Used for same purpose as PreRegister. Should be called after actual transfer of file successfully finished.

Parameters:
replicationif true, the file is being replicated between two locations registered in Indexing Service under same name.

Implements Arc::DataPoint.

Definition at line 110 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::PreRegister ( bool  replication,
bool  force = false 
) [virtual, inherited]

Index service preregistration.

This function registers the physical location of a file into an indexing service. It should be called before the actual transfer to that location happens.

Parameters:
replicationif true, the file is being replicated between two locations registered in the indexing service under same name.
forceif true, perform registration of a new file even if it already exists. Should be used to fix failures in Indexing Service.

Implements Arc::DataPoint.

Definition at line 106 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::PreUnregister ( bool  replication) [virtual, inherited]

Index Service preunregistration.

Should be called if file transfer failed. It removes changes made by PreRegister.

Parameters:
replicationif true, the file is being replicated between two locations registered in Indexing Service under same name.

Implements Arc::DataPoint.

Definition at line 114 of file DataPointDirect.cpp.

bool Arc::DataPointDirect::ProvidesMeta ( ) [virtual, inherited]

If endpoint can provide at least some meta information directly.

Implements Arc::DataPoint.

Reimplemented in Arc::DataPointGridFTP.

Definition at line 126 of file DataPointDirect.cpp.

                                     {
    return false;
  }
void Arc::DataPointDirect::Range ( unsigned long long int  start = 0,
unsigned long long int  end = 0 
) [virtual, inherited]

Set range of bytes to retrieve.

Default values correspond to whole file.

Implements Arc::DataPoint.

Definition at line 96 of file DataPointDirect.cpp.

                                                          {
    range_start = start;
    range_end = end;
  }
void Arc::DataPointFile::read_file ( ) [private]

Definition at line 71 of file DataPointFile.cpp.

                                {
    bool limit_length = false;
    unsigned long long int range_length = 0;
    unsigned long long int offset = 0;
    if (range_end > range_start) {
      range_length = range_end - range_start;
      limit_length = true;
      lseek(fd, range_start, SEEK_SET);
      offset = range_start;
    }
    else
      lseek(fd, 0, SEEK_SET);
    for (;;) {
      if (limit_length)
        if (range_length == 0)
          break;
      /* read from fd here and push to buffer */
      /* 1. claim buffer */
      int h;
      unsigned int l;
      if (!buffer->for_read(h, l, true)) {
        /* failed to get buffer - must be error or request to exit */
        buffer->error_read(true);
        break;
      }
      if (buffer->error()) {
        buffer->is_read(h, 0, 0);
        break;
      }
      /* 2. read */
      if (limit_length)
        if (l > range_length)
          l = range_length;
      unsigned long long int p = lseek(fd, 0, SEEK_CUR);
      if (p == (unsigned long long int)(-1))
        p = offset;
      int ll = read(fd, (*(buffer))[h], l);
      if (ll == -1) { /* error */
        buffer->is_read(h, 0, 0);
        buffer->error_read(true);
        break;
      }
      if (ll == 0) { /* eof */
        buffer->is_read(h, 0, 0);
        break;
      }
      /* 3. announce */
      buffer->is_read(h, ll, p);
      if (limit_length) {
        if (ll > range_length)
          range_length = 0;
        else
          range_length -= ll;
      }
      offset += ll; // for non-seakable files
    }
    close(fd);
    buffer->eof_read(true);
    transfer_cond.signal();
  }

Here is the call graph for this function:

void Arc::DataPointFile::read_file_start ( void *  arg) [static, private]

Definition at line 67 of file DataPointFile.cpp.

                                               {
    ((DataPointFile*)arg)->read_file();
  }

Here is the caller graph for this function:

bool Arc::DataPointDirect::ReadOnly ( ) const [virtual, inherited]

Implements Arc::DataPoint.

Definition at line 64 of file DataPointDirect.cpp.

                                       {
    return readonly;
  }
void Arc::DataPointDirect::ReadOutOfOrder ( bool  v) [virtual, inherited]

Allow/disallow DataPoint to produce scattered data during reading operation.

Parameters:
vtrue if allowed (default is false).

Implements Arc::DataPoint.

Definition at line 68 of file DataPointDirect.cpp.

                                               {
    allow_out_of_order = val;
  }
bool Arc::DataPointDirect::Registered ( ) const [virtual, inherited]

Check if file is registered in Indexing Service.

Proper value is obtainable only after Resolve.

Implements Arc::DataPoint.

Definition at line 130 of file DataPointDirect.cpp.

                                         {
    return false;
  }

Remove/delete object at URL.

Implements Arc::DataPoint.

Definition at line 209 of file DataPointFile.cpp.

                                   {
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsReadingError;
      
    const char* path = url.Path().c_str();
    struct stat st;
    if(stat(path,&st) != 0) {
      if (errno == ENOENT) return DataStatus::Success;
      logger.msg(INFO, "File is not accessible: %s - %s", path, strerror(errno));
      return DataStatus::DeleteError;
    }
    // path is a directory
    if(S_ISDIR(st.st_mode)) {
      if (rmdir(path) == -1) {
        logger.msg(INFO, "Can't delete directory: %s - %s", path, strerror(errno));
        return DataStatus::DeleteError;
      }
      return DataStatus::Success;
    }
    // path is a file
    if(unlink(path) == -1 && errno != ENOENT) {
      logger.msg(INFO, "Can't delete file: %s - %s", path, strerror(errno));
      return DataStatus::DeleteError;
    }
    return DataStatus::Success;
  }

Here is the call graph for this function:

Remove current URL from list.

Implements Arc::DataPoint.

Definition at line 165 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::RemoveLocations ( const DataPoint p) [virtual, inherited]

Remove locations present in another DataPoint object.

Implements Arc::DataPoint.

Definition at line 169 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::Resolve ( bool  source) [virtual, inherited]

Resolves index service URL into list of ordinary URLs.

Also obtains meta information about the file.

Parameters:
sourcetrue if DataPoint object represents source of information.

Implements Arc::DataPoint.

Definition at line 102 of file DataPointDirect.cpp.

void Arc::DataPointDirect::SetAdditionalChecks ( bool  v) [virtual, inherited]

Allow/disallow additional checks.

Check for existance of remote file (and probably other checks too) before initiating reading and writing operations.

Parameters:
vtrue if allowed (default is true).

Implements Arc::DataPoint.

Definition at line 76 of file DataPointDirect.cpp.

                                                    {
    additional_checks = val;
  }
void Arc::DataPoint::SetCheckSum ( const std::string &  val) [virtual, inherited]

Set value of meta-information 'checksum'.

Definition at line 106 of file DataPoint.cpp.

                                                  {
    checksum = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetCreated ( const Time val) [virtual, inherited]

Set value of meta-information 'creation/modification time'.

Definition at line 122 of file DataPoint.cpp.

                                            {
    created = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetMeta ( const DataPoint p) [virtual, inherited]

Copy meta information from another object.

Already defined values are not overwritten.

Parameters:
pobject from which information is taken.

Definition at line 154 of file DataPoint.cpp.

                                            {
    if (!CheckSize())
      SetSize(p.GetSize());
    if (!CheckCheckSum())
      SetCheckSum(p.GetCheckSum());
    if (!CheckCreated())
      SetCreated(p.GetCreated());
    if (!CheckValid())
      SetValid(p.GetValid());
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointDirect::SetSecure ( bool  v) [virtual, inherited]

Allow/disallow heavy security during data transfer.

Parameters:
vtrue if allowed (default depends on protocol).

Implements Arc::DataPoint.

Definition at line 84 of file DataPointDirect.cpp.

                                          {
    force_secure = val;
  }
void Arc::DataPoint::SetSize ( const unsigned long long int  val) [virtual, inherited]

Set value of meta-information 'size'.

Definition at line 94 of file DataPoint.cpp.

                                                          {
    size = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetTries ( const int  n) [virtual, inherited]

Set number of retries.

Reimplemented in Arc::DataPointIndex.

Definition at line 146 of file DataPoint.cpp.

                                      {
    triesleft = std::max(0, n);
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetValid ( const Time val) [virtual, inherited]

Set value of meta-information 'validity time'.

Definition at line 134 of file DataPoint.cpp.

                                          {
    valid = val;
  }

Here is the caller graph for this function:

Start reading data from URL.

Separate thread to transfer data will be created. No other operation can be performed while reading is in progress.

Parameters:
bufferoperation will use this buffer to put information into. Should not be destroyed before stop_reading was called and returned.

Implements Arc::DataPoint.

Definition at line 238 of file DataPointFile.cpp.

                                                        {
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    reading = true;
    /* try to open */
    int flags = O_RDONLY;
#ifdef WIN32
    flags |= O_BINARY;
#endif

    if (url.Path() == "-") // won't work
      fd = dup(STDIN_FILENO);
    else {
      User user;
      if (user.check_file_access(url.Path(), flags) != 0) {
        reading = false;
        return DataStatus::ReadStartError;
      }
      fd = open((url.Path()).c_str(), flags);
    }
    if (fd == -1) {
      reading = false;
      return DataStatus::ReadStartError;
    }
    /* provide some metadata */
    struct stat st;
    if (fstat(fd, &st) == 0) {
      SetSize(st.st_size);
      SetCreated(st.st_mtime);
    }
    buffer = &buf;
    transfer_cond.reset();
    /* create thread to maintain reading */
    if(!CreateThreadFunction(&DataPointFile::read_file_start,this)) {
      close(fd);
      fd = -1;
      reading = false;
      return DataStatus::ReadStartError;
    }
    return DataStatus::Success;
  }

Here is the call graph for this function:

DataStatus Arc::DataPointFile::StartWriting ( DataBuffer buffer,
DataCallback space_cb = NULL 
) [virtual]

Start writing data to URL.

Separate thread to transfer data will be created. No other operation can be performed while writing is in progress.

Parameters:
bufferoperation will use this buffer to get information from. Should not be destroyed before stop_writing was called and returned.
space_cbcallback which is called if there is not enough space to store data. May not implemented for all protocols.

Implements Arc::DataPoint.

Definition at line 298 of file DataPointFile.cpp.

                                                                 {
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    writing = true;
    /* try to open */
    buffer = &buf;
    if (url.Path() == "-") { // won't work
      fd = dup(STDOUT_FILENO);
      if (fd == -1) {
        logger.msg(ERROR, "Failed to use channel stdout");
        buffer->error_write(true);
        buffer->eof_write(true);
        writing = false;
        return DataStatus::WriteStartError;
      }
    }
    else {
      User user;
      /* do not check permissions to create anything here -
         suppose it path was checked at higher level */
      /* make directories */
      if (url.Path().empty()) {
        logger.msg(ERROR, "Invalid url: %s", url.str());
        buffer->error_write(true);
        buffer->eof_write(true);
        writing = false;
        return DataStatus::WriteStartError;
      }
      std::string dirpath = Glib::path_get_dirname(url.Path());
      if(dirpath == ".") dirpath = G_DIR_SEPARATOR_S; // shouldn't happen
      if (mkdir_recursive("", dirpath, S_IRWXU, user) != 0)
        if (errno != EEXIST) {
          logger.msg(ERROR, "Failed to create/find directory %s, (%d)", dirpath, errno);
          buffer->error_write(true);
          buffer->eof_write(true);
          writing = false;
          return DataStatus::WriteStartError;
        }

      /* try to create file, if failed - try to open it */
      int flags = O_WRONLY;
#ifdef WIN32
      flags |= O_BINARY;
#endif
      fd = open((url.Path()).c_str(), flags | O_CREAT | O_EXCL, S_IRUSR | S_IWUSR);
      if (fd == -1)
        fd = open((url.Path()).c_str(), flags | O_TRUNC, S_IRUSR | S_IWUSR);
      else  /* this file was created by us. Hence we can set it's owner */
        (fchown(fd, user.get_uid(), user.get_gid()) != 0);
      if (fd == -1) {
        logger.msg(ERROR, "Failed to create/open file %s (%d)", url.Path(), errno);
        buffer->error_write(true);
        buffer->eof_write(true);
        writing = false;
        return DataStatus::WriteStartError;
      }

      /* preallocate space */
      buffer->speed.hold(true);
      if (additional_checks && CheckSize()) {
        unsigned long long int fsize = GetSize();
        logger.msg(INFO, "setting file %s to size %llu",
                   url.Path(), fsize);
        /* because filesytem can skip empty blocks do real write */
        unsigned long long int old_size = lseek(fd, 0, SEEK_END);
        if (old_size < fsize) {
          char buf[65536];
          memset(buf, 0xFF, sizeof(buf));
          unsigned int l = 1;
          while (l > 0) {
            old_size = lseek(fd, 0, SEEK_END);
            l = sizeof(buf);
            if (l > (fsize - old_size))
              l = fsize - old_size;
            if (write(fd, buf, l) == -1) {
              /* out of space */
              if (space_cb != NULL)
                if (space_cb->cb((unsigned long long int)l))
                  continue;
              lseek(fd, 0, SEEK_SET);
              (ftruncate(fd, 0) != 0);
              close(fd);
              fd = -1;
              logger.msg(INFO, "Failed to preallocate space");
              buffer->speed.reset();
              buffer->speed.hold(false);
              buffer->error_write(true);
              buffer->eof_write(true);
              writing = false;
              return DataStatus::WriteStartError;
            }
          }
        }
      }
    }
    buffer->speed.reset();
    buffer->speed.hold(false);
    transfer_cond.reset();
    /* create thread to maintain writing */
    if(!CreateThreadFunction(&DataPointFile::write_file_start,this)) {
      close(fd);
      fd = -1;
      buffer->error_write(true);
      buffer->eof_write(true);
      writing = false;
      return DataStatus::WriteStartError;
    }
    return DataStatus::Success;
  }

Here is the call graph for this function:

Stop reading.

Must be called after corresponding start_reading method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred. Must return failure if any happened during transfer.

Implements Arc::DataPoint.

Definition at line 282 of file DataPointFile.cpp.

                                        {
    if (!reading)
      return DataStatus::ReadStopError;
    reading = false;
    if (!buffer->eof_read()) {
      buffer->error_read(true);      /* trigger transfer error */
      close(fd);
      fd = -1;
    }
    // buffer->wait_eof_read();
    transfer_cond.wait();         /* wait till reading thread exited */
    if (buffer->error_read())
      return DataStatus::ReadError;
    return DataStatus::Success;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Stop writing.

Must be called after corresponding start_writing method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred. Must return failure if any happened during transfer.

Implements Arc::DataPoint.

Definition at line 411 of file DataPointFile.cpp.

                                        {
    if (!writing)
      return DataStatus::WriteStopError;
    writing = false;
    if (!buffer->eof_write()) {
      buffer->error_write(true);      /* trigger transfer error */
      close(fd);
      fd = -1;
    }
    // buffer->wait_eof_write();
    transfer_cond.wait();         /* wait till writing thread exited */
    // validate file size
    if (additional_checks && CheckSize()) {
      struct stat st;
      std::string path = url.Path();
      if (stat(path.c_str(), &st) != 0 && errno != ENOENT) {
        logger.msg(ERROR, "Error during file validation. Can't stat file %s", url.Path());
        return DataStatus::WriteStopError;
      }
      if (errno != ENOENT && GetSize() != st.st_size) {
        logger.msg(ERROR, "Error during file validation: Local file size %llu does not match source file size %llu for file %s",
                          st.st_size, GetSize(), url.Path());
        return DataStatus::WriteStopError;
      }
    }
    
    if (buffer->error_write())
      return DataStatus::WriteError;
    return DataStatus::Success;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

std::string Arc::DataPoint::str ( ) const [virtual, inherited]

Returns a string representation of the DataPoint.

Definition at line 53 of file DataPoint.cpp.

                                 {
    return url.str();
  }

Here is the call graph for this function:

Here is the caller graph for this function:

DataStatus Arc::DataPointDirect::Unregister ( bool  all) [virtual, inherited]

Index Service unregistration.

Remove information about file registered in Indexing Service.

Parameters:
allif true, information about file itself is (LFN) is removed. Otherwise only particular physical instance is unregistered.

Implements Arc::DataPoint.

Definition at line 118 of file DataPointDirect.cpp.

void Arc::DataPointFile::write_file ( ) [private]

Definition at line 136 of file DataPointFile.cpp.

                                 {
    for (;;) {
      /* take from buffer and write to fd */
      /* 1. claim buffer */
      int h;
      unsigned int l;
      unsigned long long int p;
      if (!buffer->for_write(h, l, p, true)) {
        /* failed to get buffer - must be error or request to exit */
        if (!buffer->eof_read())
          buffer->error_write(true);
        buffer->eof_write(true);
        break;
      }
      if (buffer->error()) {
        buffer->is_written(h);
        buffer->eof_write(true);
        break;
      }
      /* 2. write */
      lseek(fd, p, SEEK_SET);
      int l_ = 0;
      int ll = 0;
      while (l_ < l) {
        ll = write(fd, (*(buffer))[h] + l_, l - l_);
        if (ll == -1) { /* error */
          buffer->is_written(h);
          buffer->error_write(true);
          buffer->eof_write(true);
          break;
        }
        l_ += ll;
      }
      if (ll == -1)
        break;
      /* 3. announce */
      buffer->is_written(h);
    }
#ifndef WIN32
    // This is for broken filesystems. Specifically for Lustre.
    if (fsync(fd) != 0 && errno != EINVAL) { // this error is caused by special files like stdout
      logger.msg(ERROR, "fsync of file %s failed: %s", url.Path(), strerror(errno));
      buffer->error_write(true);
    }
#endif
    if (close(fd) != 0) {
      logger.msg(ERROR, "closing file %s failed: %s", url.Path(), strerror(errno));
      buffer->error_write(true);
    }    
    transfer_cond.signal();
  }

Here is the call graph for this function:

void Arc::DataPointFile::write_file_start ( void *  arg) [static, private]

Definition at line 132 of file DataPointFile.cpp.

                                                {
    ((DataPointFile*)arg)->write_file();
  }

Here is the caller graph for this function:

Returns true if URL can accept scattered data for writing operation.

Reimplemented from Arc::DataPointDirect.

Definition at line 525 of file DataPointFile.cpp.

                                      {
    if (!url)
      return false;
    if (url.Protocol() == "file")
      return true;
    return false;
  }

Here is the call graph for this function:


Member Data Documentation

bool Arc::DataPointDirect::additional_checks [protected, inherited]

Definition at line 84 of file DataPointDirect.h.

bool Arc::DataPointDirect::allow_out_of_order [protected, inherited]

Definition at line 85 of file DataPointDirect.h.

DataBuffer* Arc::DataPointDirect::buffer [protected, inherited]

Definition at line 75 of file DataPointDirect.h.

int Arc::DataPointDirect::bufnum [protected, inherited]

Definition at line 77 of file DataPointDirect.h.

long long int Arc::DataPointDirect::bufsize [protected, inherited]

Definition at line 76 of file DataPointDirect.h.

bool Arc::DataPoint::cache [protected, inherited]

Definition at line 325 of file DataPoint.h.

std::string Arc::DataPoint::checksum [protected, inherited]

Definition at line 320 of file DataPoint.h.

Time Arc::DataPoint::created [protected, inherited]

Definition at line 321 of file DataPoint.h.

DataStatus Arc::DataPoint::failure_code [protected, inherited]

Definition at line 324 of file DataPoint.h.

int Arc::DataPointFile::fd [private]

Definition at line 38 of file DataPointFile.h.

bool Arc::DataPointDirect::force_passive [protected, inherited]

Definition at line 83 of file DataPointDirect.h.

bool Arc::DataPointDirect::force_secure [protected, inherited]

Definition at line 82 of file DataPointDirect.h.

Definition at line 39 of file DataPointFile.h.

bool Arc::DataPointDirect::is_secure [protected, inherited]

Definition at line 81 of file DataPointDirect.h.

bool Arc::DataPointDirect::linkable [protected, inherited]

Definition at line 80 of file DataPointDirect.h.

bool Arc::DataPointDirect::local [protected, inherited]

Definition at line 78 of file DataPointDirect.h.

Reimplemented from Arc::DataPoint.

Definition at line 40 of file DataPointFile.h.

unsigned long long int Arc::DataPointDirect::range_end [protected, inherited]

Definition at line 87 of file DataPointDirect.h.

unsigned long long int Arc::DataPointDirect::range_start [protected, inherited]

Definition at line 86 of file DataPointDirect.h.

Definition at line 36 of file DataPointFile.h.

bool Arc::DataPointDirect::readonly [protected, inherited]

Definition at line 79 of file DataPointDirect.h.

unsigned long long int Arc::DataPoint::size [protected, inherited]

Definition at line 319 of file DataPoint.h.

Definition at line 31 of file DataPointFile.h.

int Arc::DataPoint::triesleft [protected, inherited]

Definition at line 323 of file DataPoint.h.

const URL& Arc::DataPoint::url [protected, inherited]

Definition at line 315 of file DataPoint.h.

const UserConfig& Arc::DataPoint::usercfg [protected, inherited]

Definition at line 316 of file DataPoint.h.

Time Arc::DataPoint::valid [protected, inherited]

Definition at line 322 of file DataPoint.h.

std::list<std::string> Arc::DataPoint::valid_url_options [protected, inherited]

Subclasses should add their own specific options to this list.

Definition at line 327 of file DataPoint.h.

Definition at line 37 of file DataPointFile.h.


The documentation for this class was generated from the following files: