Back to index

nordugrid-arc-nox  1.1.0~rc6
Public Member Functions | Static Public Member Functions | Protected Attributes | Private Member Functions | Static Private Member Functions | Private Attributes | Static Private Attributes
Arc::DataPointGridFTP Class Reference

#include <DataPointGridFTP.h>

Inheritance diagram for Arc::DataPointGridFTP:
Inheritance graph
[legend]
Collaboration diagram for Arc::DataPointGridFTP:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 DataPointGridFTP (const URL &url, const UserConfig &usercfg)
virtual ~DataPointGridFTP ()
virtual DataStatus StartReading (DataBuffer &buf)
 Start reading data from URL.
virtual DataStatus StartWriting (DataBuffer &buf, DataCallback *space_cb=NULL)
 Start writing data to URL.
virtual DataStatus StopReading ()
 Stop reading.
virtual DataStatus StopWriting ()
 Stop writing.
virtual DataStatus Check ()
 Query the DataPoint to check if object is accessible.
virtual DataStatus Remove ()
 Remove/delete object at URL.
virtual DataStatus ListFiles (std::list< FileInfo > &files, bool long_list=false, bool resolve=false, bool metadata=false)
 List file(s).
virtual bool WriteOutOfOrder ()
 Returns true if URL can accept scattered data for writing operation.
virtual bool ProvidesMeta ()
 If endpoint can provide at least some meta information directly.
virtual bool IsIndex () const
 Check if URL is an Indexing Service.
virtual long long int BufSize () const
 Get suggested buffer size for transfers.
virtual int BufNum () const
 Get suggested number of buffers for transfers.
virtual bool Local () const
 Returns true if file is local, e.g. file:// urls.
virtual bool ReadOnly () const
virtual void ReadOutOfOrder (bool v)
 Allow/disallow DataPoint to produce scattered data during reading operation.
virtual void SetAdditionalChecks (bool v)
 Allow/disallow additional checks.
virtual bool GetAdditionalChecks () const
 Check if additional checks before will be performed.
virtual void SetSecure (bool v)
 Allow/disallow heavy security during data transfer.
virtual bool GetSecure () const
 Check if heavy security during data transfer is allowed.
virtual void Passive (bool v)
 Request passive transfers for FTP-like protocols.
virtual void Range (unsigned long long int start=0, unsigned long long int end=0)
 Set range of bytes to retrieve.
virtual DataStatus Resolve (bool source)
 Resolves index service URL into list of ordinary URLs.
virtual bool Registered () const
 Check if file is registered in Indexing Service.
virtual DataStatus PreRegister (bool replication, bool force=false)
 Index service preregistration.
virtual DataStatus PostRegister (bool replication)
 Index Service postregistration.
virtual DataStatus PreUnregister (bool replication)
 Index Service preunregistration.
virtual DataStatus Unregister (bool all)
 Index Service unregistration.
virtual bool AcceptsMeta ()
 If endpoint can have any use from meta information.
virtual const URLCurrentLocation () const
 Returns current (resolved) URL.
virtual const std::string & CurrentLocationMetadata () const
 Returns meta information used to create current URL.
virtual DataStatus CompareLocationMetadata () const
 Compare metadata of DataPoint and current location.
virtual bool NextLocation ()
 Switch to next location in list of URLs.
virtual bool LocationValid () const
 Returns false if out of retries.
virtual bool HaveLocations () const
 Returns true if number of resolved URLs is not 0.
virtual DataStatus AddLocation (const URL &url, const std::string &meta)
 Add URL to list.
virtual DataStatus RemoveLocation ()
 Remove current URL from list.
virtual DataStatus RemoveLocations (const DataPoint &p)
 Remove locations present in another DataPoint object.
virtual const URLGetURL () const
 Returns the URL that was passed to the constructor.
virtual const UserConfigGetUserConfig () const
 Returns the UserConfig that was passed to the constructor.
virtual std::string str () const
 Returns a string representation of the DataPoint.
virtual operator bool () const
 Is DataPoint valid?
virtual bool operator! () const
 Is DataPoint valid?
virtual DataStatus GetFailureReason (void) const
 Returns reason of transfer failure, as reported by callbacks. This could be different from the failure returned by the methods themselves.
virtual bool CheckSize () const
 Check if meta-information 'size' is available.
virtual void SetSize (const unsigned long long int val)
 Set value of meta-information 'size'.
virtual unsigned long long int GetSize () const
 Get value of meta-information 'size'.
virtual bool CheckCheckSum () const
 Check if meta-information 'checksum' is available.
virtual void SetCheckSum (const std::string &val)
 Set value of meta-information 'checksum'.
virtual const std::string & GetCheckSum () const
 Get value of meta-information 'checksum'.
virtual const std::string DefaultCheckSum () const
 Default checksum type.
virtual bool CheckCreated () const
 Check if meta-information 'creation/modification time' is available.
virtual void SetCreated (const Time &val)
 Set value of meta-information 'creation/modification time'.
virtual const TimeGetCreated () const
 Get value of meta-information 'creation/modification time'.
virtual bool CheckValid () const
 Check if meta-information 'validity time' is available.
virtual void SetValid (const Time &val)
 Set value of meta-information 'validity time'.
virtual const TimeGetValid () const
 Get value of meta-information 'validity time'.
virtual bool Cache () const
 Returns true if file is cacheable.
virtual int GetTries () const
 Returns number of retries left.
virtual void SetTries (const int n)
 Set number of retries.
virtual void NextTry (void)
 Decrease number of retries left.
virtual void SetMeta (const DataPoint &p)
 Copy meta information from another object.
virtual bool CompareMeta (const DataPoint &p) const
 Compare meta information from another object.

Static Public Member Functions

static PluginInstance (PluginArgument *arg)

Protected Attributes

DataBufferbuffer
long long int bufsize
int bufnum
bool local
bool readonly
bool linkable
bool is_secure
bool force_secure
bool force_passive
bool additional_checks
bool allow_out_of_order
unsigned long long int range_start
unsigned long long int range_end
const URLurl
const UserConfigusercfg
unsigned long long int size
std::string checksum
Time created
Time valid
int triesleft
DataStatus failure_code
bool cache
std::list< std::string > valid_url_options
 Subclasses should add their own specific options to this list.

Private Member Functions

bool mkdir_ftp ()
bool check_credentials ()
void set_attributes ()

Static Private Member Functions

static void ftp_complete_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error)
static void ftp_get_complete_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error)
static void ftp_put_complete_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error)
static void ftp_read_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error, globus_byte_t *buffer, globus_size_t length, globus_off_t offset, globus_bool_t eof)
static void ftp_check_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error, globus_byte_t *buffer, globus_size_t length, globus_off_t offset, globus_bool_t eof)
static void ftp_write_callback (void *arg, globus_ftp_client_handle_t *handle, globus_object_t *error, globus_byte_t *buffer, globus_size_t length, globus_off_t offset, globus_bool_t eof)
static void * ftp_read_thread (void *arg)
static void * ftp_write_thread (void *arg)

Private Attributes

bool ftp_active
globus_ftp_client_handle_t ftp_handle
globus_ftp_client_operationattr_t ftp_opattr
globus_thread_t ftp_control_thread
int ftp_threads
bool autodir
SimpleCondition cond
DataStatus condstatus
GSSCredentialcredential
bool reading
bool writing
bool ftp_eof_flag
std::string ftp_dir_path
char ftp_buf [16]

Static Private Attributes

static Logger logger

Detailed Description

Definition at line 20 of file DataPointGridFTP.h.


Constructor & Destructor Documentation

Arc::DataPointGridFTP::DataPointGridFTP ( const URL url,
const UserConfig usercfg 
)

Definition at line 656 of file DataPointGridFTP.cpp.

    : DataPointDirect(url, usercfg),
      ftp_active(false),
      condstatus(DataStatus::Success),
      credential(NULL),
      reading(false),
      writing(false) {
    //globus_module_activate(GLOBUS_FTP_CLIENT_MODULE);
    //if (!proxy_initialized)
    //  proxy_initialized = GlobusRecoverProxyOpenSSL();
    // Activating globus only once because it looks like 
    // deactivation of GLOBUS_FTP_CONTROL_MODULE is not
    // handled properly on Windows. This should not cause
    // problems (except for valgrind) because this plugin
    // is registered as persistent.
    if (!proxy_initialized) {
      globus_module_activate(GLOBUS_COMMON_MODULE);
      globus_module_activate(GLOBUS_FTP_CLIENT_MODULE);
      proxy_initialized = GlobusRecoverProxyOpenSSL();
    }
    is_secure = false;
    if (url.Protocol() == "gsiftp")
      is_secure = true;
    if (!ftp_active) {
      GlobusResult res;
      globus_ftp_client_handleattr_t ftp_attr;
      if (!(res = globus_ftp_client_handleattr_init(&ftp_attr))) {
        logger.msg(ERROR,
                   "init_handle: globus_ftp_client_handleattr_init failed");
        logger.msg(ERROR, "Globus error: %s", res.str());
        ftp_active = false;
        return;
      }
#ifdef HAVE_GLOBUS_FTP_CLIENT_HANDLEATTR_SET_GRIDFTP2
      if (!(res = globus_ftp_client_handleattr_set_gridftp2(&ftp_attr,
                                                            GLOBUS_TRUE))) {
        globus_ftp_client_handleattr_destroy(&ftp_attr);
        logger.msg(ERROR, "init_handle: "
                   "globus_ftp_client_handleattr_set_gridftp2 failed");
        logger.msg(ERROR, "Globus error: %s", res.str());
        ftp_active = false;
        return;
      }
#endif
      if (!(res = globus_ftp_client_handle_init(&ftp_handle, &ftp_attr))) {
        globus_ftp_client_handleattr_destroy(&ftp_attr);
        logger.msg(ERROR, "init_handle: globus_ftp_client_handle_init failed");
        logger.msg(ERROR, "Globus error: %s", res.str());
        ftp_active = false;
        return;
      }
      globus_ftp_client_handleattr_destroy(&ftp_attr);
      if (!(res = globus_ftp_client_operationattr_init(&ftp_opattr))) {
        logger.msg(ERROR, "init_handle: "
                   "globus_ftp_client_operationattr_init failed");
        logger.msg(ERROR, "Globus error: %s", res.str());
        globus_ftp_client_handle_destroy(&ftp_handle);
        ftp_active = false;
        return;
      }
    }
    ftp_active = true;
    ftp_threads = 1;
    if (allow_out_of_order) {
      ftp_threads = stringtoi(url.Option("threads"));
      if (ftp_threads < 1)
        ftp_threads = 1;
      if (ftp_threads > MAX_PARALLEL_STREAMS)
        ftp_threads = MAX_PARALLEL_STREAMS;
    }
    autodir = additional_checks;
    std::string autodir_s = url.Option("autodir");
    if(autodir_s == "yes") {
      autodir = true;
    } else if(autodir_s == "no") {
      autodir = false;
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Definition at line 805 of file DataPointGridFTP.cpp.

                                      {
    StopReading();
    StopWriting();
    if (ftp_active) {
      logger.msg(VERBOSE, "DataPoint::deinit_handle: destroy ftp_handle");
      globus_ftp_client_handle_destroy(&ftp_handle);
      globus_ftp_client_operationattr_destroy(&ftp_opattr);
    }
    if (credential)
      delete credential;
    // See activation for description
    //globus_module_deactivate(GLOBUS_FTP_CLIENT_MODULE);
  }

Here is the call graph for this function:


Member Function Documentation

bool Arc::DataPointDirect::AcceptsMeta ( ) [virtual, inherited]

If endpoint can have any use from meta information.

Implements Arc::DataPoint.

Definition at line 122 of file DataPointDirect.cpp.

                                    {
    return false;
  }
DataStatus Arc::DataPointDirect::AddLocation ( const URL url,
const std::string &  meta 
) [virtual, inherited]

Add URL to list.

Parameters:
urlLocation URL to add.
metaLocation meta information.

Implements Arc::DataPoint.

Definition at line 161 of file DataPointDirect.cpp.

int Arc::DataPointDirect::BufNum ( ) const [virtual, inherited]

Get suggested number of buffers for transfers.

Implements Arc::DataPoint.

Definition at line 56 of file DataPointDirect.cpp.

                                    {
    return bufnum;
  }
long long int Arc::DataPointDirect::BufSize ( ) const [virtual, inherited]

Get suggested buffer size for transfers.

Implements Arc::DataPoint.

Definition at line 52 of file DataPointDirect.cpp.

                                               {
    return bufsize;
  }
bool Arc::DataPoint::Cache ( ) const [virtual, inherited]

Returns true if file is cacheable.

Definition at line 86 of file DataPoint.cpp.

                              {
    return cache;
  }

Here is the caller graph for this function:

Query the DataPoint to check if object is accessible.

If possible this method will also try to provide meta information about the object.

Implements Arc::DataPoint.

Definition at line 74 of file DataPointGridFTP.cpp.

                                     {
    if (!ftp_active)
      return DataStatus::NotInitializedError;
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    GlobusResult res;
    globus_off_t size = 0;
    globus_abstime_t gl_modify_time;
    time_t modify_time;
    int modify_utime;
    set_attributes();
    res = globus_ftp_client_size(&ftp_handle, url.str().c_str(), &ftp_opattr,
                                 &size, &ftp_complete_callback, this);
    if (!res) {
      logger.msg(VERBOSE, "check_ftp: globus_ftp_client_size failed");
      logger.msg(INFO, "Globus error: %s", res.str());
    }
    else if (!cond.wait(1000*usercfg.Timeout())) {
      logger.msg(INFO, "check_ftp: timeout waiting for size");
      globus_ftp_client_abort(&ftp_handle);
      cond.wait();
    }
    else if (!condstatus)
      logger.msg(INFO, "check_ftp: failed to get file's size");
    else {
      SetSize(size);
      logger.msg(VERBOSE, "check_ftp: obtained size: %lli", GetSize());
    }
    res = globus_ftp_client_modification_time(&ftp_handle, url.str().c_str(),
                                              &ftp_opattr, &gl_modify_time,
                                              &ftp_complete_callback, this);
    if (!res) {
      logger.msg(VERBOSE,
                 "check_ftp: globus_ftp_client_modification_time failed");
      logger.msg(INFO, "Globus error: %s", res.str());
    }
    else if (!cond.wait(1000*usercfg.Timeout())) {
      logger.msg(INFO, "check_ftp: timeout waiting for modification_time");
      globus_ftp_client_abort(&ftp_handle);
      cond.wait();
    }
    else if (!condstatus)
      logger.msg(INFO, "check_ftp: failed to get file's modification time");
    else {
      GlobusTimeAbstimeGet(gl_modify_time, modify_time, modify_utime);
      SetCreated(modify_time);
      logger.msg(VERBOSE, "check_ftp: obtained creation date: %s", GetCreated().str());
    }
    // Do not use partial_get for ordinary ftp. Stupid globus tries to
    // use non-standard commands anyway.
    if (is_secure) {
      res = globus_ftp_client_partial_get(&ftp_handle, url.str().c_str(),
                                          &ftp_opattr, GLOBUS_NULL, 0, 1,
                                          &ftp_complete_callback, this);
      if (!res) {
        logger.msg(VERBOSE, "check_ftp: globus_ftp_client_get failed");
        logger.msg(INFO, "Globus error: %s", res.str());
        return DataStatus::CheckError;
      }
      // use eof_flag to pass result from callback
      ftp_eof_flag = false;
      logger.msg(VERBOSE, "check_ftp: globus_ftp_client_register_read");
      res = globus_ftp_client_register_read(&ftp_handle,
                                            (globus_byte_t*)ftp_buf,
                                            sizeof(ftp_buf),
                                            &ftp_check_callback, this);
      if (!res) {
        globus_ftp_client_abort(&ftp_handle);
        cond.wait();
        return DataStatus::CheckError;
      }
      if (!cond.wait(1000*usercfg.Timeout())) {
        logger.msg(INFO, "check_ftp: timeout waiting for partial get");
        globus_ftp_client_abort(&ftp_handle);
        cond.wait();
        return DataStatus::CheckError;
      }
      return condstatus;
    }
    else {
      // Do not use it at all. It does not give too much useful
      // information anyway. But request at least existence of file.
      if (!CheckSize())
        return DataStatus::CheckError;
      return DataStatus::Success;
    }
  }

Here is the call graph for this function:

bool Arc::DataPoint::CheckCheckSum ( ) const [virtual, inherited]

Check if meta-information 'checksum' is available.

Definition at line 102 of file DataPoint.cpp.

                                      {
    return (!checksum.empty());
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckCreated ( ) const [virtual, inherited]

Check if meta-information 'creation/modification time' is available.

Definition at line 118 of file DataPoint.cpp.

                                     {
    return (created != -1);
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckSize ( ) const [virtual, inherited]

Check if meta-information 'size' is available.

Definition at line 90 of file DataPoint.cpp.

                                  {
    return (size != (unsigned long long int)(-1));
  }

Here is the caller graph for this function:

bool Arc::DataPoint::CheckValid ( ) const [virtual, inherited]

Check if meta-information 'validity time' is available.

Definition at line 130 of file DataPoint.cpp.

                                   {
    return (valid != -1);
  }

Here is the caller graph for this function:

Compare metadata of DataPoint and current location.

Returns inconsistency error or error encountered during operation, or success

Implements Arc::DataPoint.

Definition at line 143 of file DataPointDirect.cpp.

bool Arc::DataPoint::CompareMeta ( const DataPoint p) const [virtual, inherited]

Compare meta information from another object.

Undefined values are not used for comparison.

Parameters:
pobject to which to compare.

Definition at line 165 of file DataPoint.cpp.

                                                      {
    if (CheckSize() && p.CheckSize())
      if (GetSize() != p.GetSize())
        return false;
    if (CheckCheckSum() && p.CheckCheckSum())
      // TODO: compare checksums properly
      if (strcasecmp(GetCheckSum().c_str(), p.GetCheckSum().c_str()))
        return false;
    if (CheckValid() && p.CheckValid())
      if (GetValid() != p.GetValid())
        return false;
    return true;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

const URL & Arc::DataPointDirect::CurrentLocation ( ) const [virtual, inherited]

Returns current (resolved) URL.

Implements Arc::DataPoint.

Definition at line 134 of file DataPointDirect.cpp.

                                                    {
    return url;
  }

Here is the caller graph for this function:

const std::string & Arc::DataPointDirect::CurrentLocationMetadata ( ) const [virtual, inherited]

Returns meta information used to create current URL.

Usage differs between different indexing services.

Implements Arc::DataPoint.

Definition at line 138 of file DataPointDirect.cpp.

                                                                  {
    static const std::string empty;
    return empty;
  }
const std::string Arc::DataPoint::DefaultCheckSum ( ) const [virtual, inherited]

Default checksum type.

Reimplemented in Arc::DataPointSRM, and Arc::DataPointLFC.

Definition at line 114 of file DataPoint.cpp.

                                                   {
    return std::string("cksum");
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_check_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error,
globus_byte_t *  buffer,
globus_size_t  length,
globus_off_t  offset,
globus_bool_t  eof 
) [static, private]

Definition at line 44 of file DataPointGridFTP.cpp.

                                                               {
    logger.msg(VERBOSE, "ftp_check_callback");
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    if (error != GLOBUS_SUCCESS) {
      logger.msg(VERBOSE, "Globus error: %s", globus_object_to_string(error));
      return;
    }
    if (eof)
      return;
    GlobusResult res =
      globus_ftp_client_register_read(&(it->ftp_handle),
                                      (globus_byte_t*)(it->ftp_buf),
                                      sizeof(it->ftp_buf),
                                      &ftp_check_callback, it);
    if (!res) {
      logger.msg(INFO,
                 "Registration of Globus FTP buffer failed - cancel check");
      logger.msg(VERBOSE, "Globus error: %s", res.str());
      globus_ftp_client_abort(&(it->ftp_handle));
      return;
    }
    return;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_complete_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error 
) [static, private]

Definition at line 27 of file DataPointGridFTP.cpp.

                                                                       {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    if (error == GLOBUS_SUCCESS) {
      logger.msg(DEBUG, "ftp_complete_callback: success");
      it->condstatus = DataStatus::Success;
      it->cond.signal();
    }
    else {
      logger.msg(VERBOSE, "ftp_complete_callback: error: %s",
                 globus_object_to_string(error));
      it->condstatus = DataStatus::TransferError;
      it->cond.signal();
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_get_complete_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error 
) [static, private]

Definition at line 392 of file DataPointGridFTP.cpp.

                                                                           {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    /* data transfer finished */
    if (error != GLOBUS_SUCCESS) {
      logger.msg(INFO, "Failed to get ftp file");
      logger.msg(VERBOSE, "Globus error: %s", globus_object_to_string(error));
      it->failure_code = DataStatus(DataStatus::ReadStartError, globus_object_to_string(error));
      it->buffer->error_read(true);
      return;
    }
    it->buffer->eof_read(true);
    return;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_put_complete_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error 
) [static, private]

Definition at line 541 of file DataPointGridFTP.cpp.

                                                                           {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    /* data transfer finished */
    if (error != GLOBUS_SUCCESS) {
      logger.msg(INFO, "Failed to store ftp file");
      it->failure_code = DataStatus(DataStatus::WriteStartError, globus_object_to_string(error));
      logger.msg(VERBOSE, "Globus error: %s", globus_object_to_string(error));
      it->buffer->error_write(true);
      return;
    }
    it->buffer->eof_write(true);
    return;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_read_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error,
globus_byte_t *  buffer,
globus_size_t  length,
globus_off_t  offset,
globus_bool_t  eof 
) [static, private]

Definition at line 372 of file DataPointGridFTP.cpp.

                                                              {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    if (error != GLOBUS_SUCCESS) {
      logger.msg(VERBOSE, "ftp_read_callback: failure");
      it->buffer->is_read((char*)buffer, 0, 0);
      return;
    }
    logger.msg(DEBUG, "ftp_read_callback: success");
    it->buffer->is_read((char*)buffer, length, offset);
    if (eof)
      it->ftp_eof_flag = true;
    return;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void * Arc::DataPointGridFTP::ftp_read_thread ( void *  arg) [static, private]

Definition at line 316 of file DataPointGridFTP.cpp.

                                                   {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    int h;
    unsigned int l;
    GlobusResult res;
    int registration_failed = 0;
    logger.msg(INFO, "ftp_read_thread: get and register buffers");
    int n_buffers = 0;
    for (;;) {
      if (it->buffer->eof_read())
        break;
      if (!it->buffer->for_read(h, l, true)) { /* eof or error */
        if (it->buffer->error()) { /* error -> abort reading */
          logger.msg(VERBOSE, "ftp_read_thread: for_read failed - aborting: %s",
                     it->url.str());
          globus_ftp_client_abort(&(it->ftp_handle));
        }
        break;
      }
      res =
        globus_ftp_client_register_read(&(it->ftp_handle),
                                        (globus_byte_t*)((*(it->buffer))[h]),
                                        l, &(it->ftp_read_callback), it);
      if (!res) {
        logger.msg(DEBUG, "ftp_read_thread: Globus error: %s", res.str());
        registration_failed++;
        if (registration_failed >= 10) {
          it->buffer->is_read(h, 0, 0);
          it->buffer->error_read(true);
          // can set eof here because no callback will be called (I guess).
          it->buffer->eof_read(true);
          logger.msg(DEBUG, "ftp_read_thread: "
                     "too many registration failures - abort: %s",
                     it->url.str());
        }
        else {
          logger.msg(DEBUG, "ftp_read_thread: "
                     "failed to register globus buffer - will try later: %s",
                     it->url.str());
          it->buffer->is_read(h, 0, 0);
          sleep(1);
        }
      }
      else
        n_buffers++;
    }
    /* make sure complete callback is called */
    logger.msg(VERBOSE, "ftp_read_thread: waiting for eof");
    it->buffer->wait_eof_read();
    logger.msg(VERBOSE, "ftp_read_thread: exiting");
    it->condstatus = it->buffer->error_read() ? DataStatus::ReadError :
                     DataStatus::Success;
    it->cond.signal();
    return NULL;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointGridFTP::ftp_write_callback ( void *  arg,
globus_ftp_client_handle_t *  handle,
globus_object_t *  error,
globus_byte_t *  buffer,
globus_size_t  length,
globus_off_t  offset,
globus_bool_t  eof 
) [static, private]

Definition at line 523 of file DataPointGridFTP.cpp.

                                                           {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    if (error != GLOBUS_SUCCESS) {
      logger.msg(VERBOSE, "ftp_write_callback: failure");
      it->buffer->is_written((char*)buffer);
      return;
    }
    logger.msg(DEBUG, "ftp_write_callback: success");
    it->buffer->is_written((char*)buffer);
    return;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void * Arc::DataPointGridFTP::ftp_write_thread ( void *  arg) [static, private]

Definition at line 480 of file DataPointGridFTP.cpp.

                                                    {
    DataPointGridFTP *it = (DataPointGridFTP*)arg;
    int h;
    unsigned int l;
    unsigned long long int o;
    GlobusResult res;
    globus_bool_t eof = GLOBUS_FALSE;
    logger.msg(INFO, "ftp_write_thread: get and register buffers");
    for (;;) {
      if (!it->buffer->for_write(h, l, o, true)) {
        if (it->buffer->error()) {
          logger.msg(VERBOSE, "ftp_write_thread: for_write failed - aborting");
          globus_ftp_client_abort(&(it->ftp_handle));
          break;
        }
        // no buffers and no errors - must be pure eof
        eof = GLOBUS_TRUE;
        char dummy;
        o = it->buffer->eof_position();
        res = globus_ftp_client_register_write(&(it->ftp_handle),
                                               (globus_byte_t*)(&dummy), 0, o,
                                               eof, &ftp_write_callback, it);
        break;
        // if(res == GLOBUS_SUCCESS) break;
        // sleep(1); continue;
      }
      res =
        globus_ftp_client_register_write(&(it->ftp_handle),
                                         (globus_byte_t*)((*(it->buffer))[h]),
                                         l, o, eof, &ftp_write_callback, it);
      if (!res) {
        it->buffer->is_notwritten(h);
        sleep(1);
      }
    }
    /* make sure complete callback is called */
    it->buffer->wait_eof_write();
    it->condstatus = it->buffer->error_write() ? DataStatus::WriteError :
                     DataStatus::Success;
    it->cond.signal();
    return NULL;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

bool Arc::DataPointDirect::GetAdditionalChecks ( ) const [virtual, inherited]

Check if additional checks before will be performed.

Implements Arc::DataPoint.

Definition at line 80 of file DataPointDirect.cpp.

                                                  {
    return additional_checks;
  }
const std::string & Arc::DataPoint::GetCheckSum ( ) const [virtual, inherited]

Get value of meta-information 'checksum'.

Definition at line 110 of file DataPoint.cpp.

                                                {
    return checksum;
  }

Here is the caller graph for this function:

const Time & Arc::DataPoint::GetCreated ( ) const [virtual, inherited]

Get value of meta-information 'creation/modification time'.

Definition at line 126 of file DataPoint.cpp.

                                          {
    return created;
  }

Here is the caller graph for this function:

DataStatus Arc::DataPoint::GetFailureReason ( void  ) const [virtual, inherited]

Returns reason of transfer failure, as reported by callbacks. This could be different from the failure returned by the methods themselves.

Definition at line 82 of file DataPoint.cpp.

                                               {
    return failure_code;
  }

Here is the caller graph for this function:

bool Arc::DataPointDirect::GetSecure ( ) const [virtual, inherited]

Check if heavy security during data transfer is allowed.

Implements Arc::DataPoint.

Definition at line 88 of file DataPointDirect.cpp.

                                        {
    return is_secure;
  }
unsigned long long int Arc::DataPoint::GetSize ( ) const [virtual, inherited]

Get value of meta-information 'size'.

Definition at line 98 of file DataPoint.cpp.

                                                  {
    return size;
  }

Here is the caller graph for this function:

int Arc::DataPoint::GetTries ( ) const [virtual, inherited]

Returns number of retries left.

Definition at line 142 of file DataPoint.cpp.

                                {
    return triesleft;
  }

Here is the caller graph for this function:

const URL & Arc::DataPoint::GetURL ( ) const [virtual, inherited]

Returns the URL that was passed to the constructor.

Definition at line 45 of file DataPoint.cpp.

                                     {
    return url;
  }

Here is the caller graph for this function:

const UserConfig & Arc::DataPoint::GetUserConfig ( ) const [virtual, inherited]

Returns the UserConfig that was passed to the constructor.

Definition at line 49 of file DataPoint.cpp.

                                                   {
    return usercfg;
  }

Here is the caller graph for this function:

const Time & Arc::DataPoint::GetValid ( ) const [virtual, inherited]

Get value of meta-information 'validity time'.

Definition at line 138 of file DataPoint.cpp.

                                        {
    return valid;
  }

Here is the caller graph for this function:

bool Arc::DataPointDirect::HaveLocations ( ) const [virtual, inherited]

Returns true if number of resolved URLs is not 0.

Implements Arc::DataPoint.

Definition at line 157 of file DataPointDirect.cpp.

                                            {
    return true;
  }

Definition at line 819 of file DataPointGridFTP.cpp.

                                                        {
    DataPointPluginArgument *dmcarg = dynamic_cast<DataPointPluginArgument*>(arg);
    if (!dmcarg)
      return NULL;
    if (((const URL&)(*dmcarg)).Protocol() != "gsiftp" &&
        ((const URL&)(*dmcarg)).Protocol() != "ftp")
      return NULL;
    // Make this code non-unloadable because both OpenSSL
    // and Globus have problems with unloading
    Glib::Module* module = dmcarg->get_module();
    PluginsFactory* factory = dmcarg->get_factory();
    if(!(factory && module)) {
      logger.msg(ERROR, "Missing reference to factory and/or module. It is unsafe to use Globus in non-persistent mode - (Grid)FTP code is disabled. Report to developers.");
      return NULL;
    }
    factory->makePersistent(module);
    OpenSSLInit();
    return new DataPointGridFTP(*dmcarg, *dmcarg);
  }

Here is the call graph for this function:

bool Arc::DataPointDirect::IsIndex ( ) const [virtual, inherited]

Check if URL is an Indexing Service.

Implements Arc::DataPoint.

Definition at line 48 of file DataPointDirect.cpp.

                                      {
    return false;
  }
DataStatus Arc::DataPointGridFTP::ListFiles ( std::list< FileInfo > &  files,
bool  long_list = false,
bool  resolve = false,
bool  metadata = false 
) [virtual]

List file(s).

If the DataPoint represents a directory its contents will be listed.

Parameters:
fileswill contain list of file names and optionally their attributes.
long)listif true, list additional properties of each file.
resolveif true, resolve physical locations (relevant for indexing services only).

Implements Arc::DataPoint.

Definition at line 557 of file DataPointGridFTP.cpp.

                                                        {
    if (!ftp_active)
      return DataStatus::NotInitializedError;
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    set_attributes();
    Lister lister(*credential);
    if (lister.retrieve_dir_info(url,!(long_list | resolve | metadata)) != 0) {
      if (lister.retrieve_file_info(url,!(long_list | resolve | metadata)) != 0) {
        logger.msg(ERROR, "Failed to obtain listing from ftp: %s", url.str());
        return DataStatus::ListError;
      }
    }
    lister.close_connection();
    DataStatus result = DataStatus::Success;
    for (std::list<FileInfo>::iterator i = lister.begin();
         i != lister.end(); ++i) {
      std::list<FileInfo>::iterator f =
        files.insert(files.end(), FileInfo(i->GetLastName()));
      if (long_list) {
        GlobusResult res;
        globus_off_t size = 0;
        globus_abstime_t gl_modify_time;
        time_t modify_time;
        int modify_utime;
        // Lister should always return full path to file
        std::string f_url = url.ConnectionURL() + i->GetName();
        f->SetType(i->GetType());
        if (i->CheckSize())
          f->SetSize(i->GetSize());
        else if (i->GetType() != FileInfo::file_type_dir) {
          logger.msg(DEBUG, "list_files_ftp: looking for size of %s", f_url);
          res = globus_ftp_client_size(&ftp_handle, f_url.c_str(), &ftp_opattr,
                                       &size, &ftp_complete_callback, this);
          if (!res) {
            logger.msg(VERBOSE, "list_files_ftp: globus_ftp_client_size failed");
            logger.msg(INFO, "Globus error: %s", res.str());
            result = DataStatus::ListError;
          }
          else if (!cond.wait(1000*usercfg.Timeout())) {
            logger.msg(INFO, "list_files_ftp: timeout waiting for size");
            globus_ftp_client_abort(&ftp_handle);
            cond.wait();
            result = DataStatus::ListError;
          }
          else if (!condstatus) {
            logger.msg(INFO, "list_files_ftp: failed to get file's size");
            result = DataStatus::ListError;
            // Guessing - directories usually have no size
            f->SetType(FileInfo::file_type_dir);
          }
          else {
            f->SetSize(size);
            // Guessing - only files usually have size
            f->SetType(FileInfo::file_type_file);
          }
        }
        if (i->CheckCreated())
          f->SetCreated(i->GetCreated());
        else {
          logger.msg(DEBUG, "list_files_ftp: "
                     "looking for modification time of %s", f_url);
          res =
            globus_ftp_client_modification_time(&ftp_handle, f_url.c_str(),
                                                &ftp_opattr, &gl_modify_time,
                                                &ftp_complete_callback, this);
          if (!res) {
            logger.msg(VERBOSE, "list_files_ftp: "
                       "globus_ftp_client_modification_time failed");
            logger.msg(INFO, "Globus error: %s", res.str());
            result = DataStatus::ListError;
          }
          else if (!cond.wait(1000*usercfg.Timeout())) {
            logger.msg(INFO, "list_files_ftp: "
                       "timeout waiting for modification_time");
            globus_ftp_client_abort(&ftp_handle);
            cond.wait();
            result = DataStatus::ListError;
          }
          else if (!condstatus) {
            logger.msg(INFO, "list_files_ftp: "
                       "failed to get file's modification time");
            result = DataStatus::ListError;
          }
          else {
            GlobusTimeAbstimeGet(gl_modify_time, modify_time, modify_utime);
            f->SetCreated(modify_time);
          }
        }
      }
    }
    return result;
  }

Here is the call graph for this function:

bool Arc::DataPointDirect::Local ( ) const [virtual, inherited]

Returns true if file is local, e.g. file:// urls.

Implements Arc::DataPoint.

Definition at line 60 of file DataPointDirect.cpp.

                                    {
    return local;
  }
bool Arc::DataPointDirect::LocationValid ( ) const [virtual, inherited]

Returns false if out of retries.

Implements Arc::DataPoint.

Definition at line 153 of file DataPointDirect.cpp.

                                            {
    return (triesleft > 0);
  }

Definition at line 217 of file DataPointGridFTP.cpp.

                                   {
    ftp_dir_path = url.str();
    for (;;)
      if (!remove_last_dir(ftp_dir_path))
        break;
    bool result = false;
    for (;;) {
      if (!add_last_dir(ftp_dir_path, url.str()))
        break;
      logger.msg(VERBOSE, "mkdir_ftp: making %s", ftp_dir_path);
      GlobusResult res =
        globus_ftp_client_mkdir(&ftp_handle, ftp_dir_path.c_str(), &ftp_opattr,
                                &ftp_complete_callback, this);
      if (!res) {
        logger.msg(INFO, "Globus error: %s", res.str());
        return false;
      }
      if (!cond.wait(1000*usercfg.Timeout())) {
        logger.msg(INFO, "mkdir_ftp: timeout waiting for mkdir");
        /* timeout - have to cancel operation here */
        globus_ftp_client_abort(&ftp_handle);
        cond.wait();
        return false;
      }
      if (!condstatus)
        result = condstatus;
    }
    return result;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

bool Arc::DataPointDirect::NextLocation ( ) [virtual, inherited]

Switch to next location in list of URLs.

At last location switch to first if number of allowed retries is not exceeded. Returns false if no retries left.

Implements Arc::DataPoint.

Definition at line 147 of file DataPointDirect.cpp.

                                     {
    if (triesleft > 0)
      --triesleft;
    return (triesleft > 0);
  }
void Arc::DataPoint::NextTry ( void  ) [virtual, inherited]

Decrease number of retries left.

Definition at line 150 of file DataPoint.cpp.

                          {
    if(triesleft) --triesleft;
  }

Here is the caller graph for this function:

Arc::DataPoint::operator bool ( void  ) const [virtual, inherited]

Is DataPoint valid?

Definition at line 57 of file DataPoint.cpp.

                                 {

    if (!url)
      return false;
      
    // URL option validation. Subclasses which do not want to validate
    // URL options should override this method.
    std::map<std::string, std::string> options = url.Options();
    for (std::map<std::string, std::string>::iterator i = options.begin(); i != options.end(); i++) {
      bool valid = false; 
      for (std::list<std::string>::const_iterator j = valid_url_options.begin(); j != valid_url_options.end(); j++) {
        if (i->first == *j) valid = true;
      }
      if (!valid) {
        logger.msg(ERROR, "Invalid URL option: %s", i->first);
        return false;
      }
    }
    return true;
  }

Here is the call graph for this function:

bool Arc::DataPoint::operator! ( void  ) const [virtual, inherited]

Is DataPoint valid?

Definition at line 78 of file DataPoint.cpp.

                                  {
    return !((bool)*this);
  }
void Arc::DataPointDirect::Passive ( bool  v) [virtual, inherited]

Request passive transfers for FTP-like protocols.

Parameters:
trueto request.

Implements Arc::DataPoint.

Definition at line 92 of file DataPointDirect.cpp.

                                        {
    force_passive = val;
  }
DataStatus Arc::DataPointDirect::PostRegister ( bool  replication) [virtual, inherited]

Index Service postregistration.

Used for same purpose as PreRegister. Should be called after actual transfer of file successfully finished.

Parameters:
replicationif true, the file is being replicated between two locations registered in Indexing Service under same name.

Implements Arc::DataPoint.

Definition at line 110 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::PreRegister ( bool  replication,
bool  force = false 
) [virtual, inherited]

Index service preregistration.

This function registers the physical location of a file into an indexing service. It should be called before the actual transfer to that location happens.

Parameters:
replicationif true, the file is being replicated between two locations registered in the indexing service under same name.
forceif true, perform registration of a new file even if it already exists. Should be used to fix failures in Indexing Service.

Implements Arc::DataPoint.

Definition at line 106 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::PreUnregister ( bool  replication) [virtual, inherited]

Index Service preunregistration.

Should be called if file transfer failed. It removes changes made by PreRegister.

Parameters:
replicationif true, the file is being replicated between two locations registered in Indexing Service under same name.

Implements Arc::DataPoint.

Definition at line 114 of file DataPointDirect.cpp.

If endpoint can provide at least some meta information directly.

Reimplemented from Arc::DataPointDirect.

Definition at line 843 of file DataPointGridFTP.cpp.

                                      {
    return true;
  }
void Arc::DataPointDirect::Range ( unsigned long long int  start = 0,
unsigned long long int  end = 0 
) [virtual, inherited]

Set range of bytes to retrieve.

Default values correspond to whole file.

Implements Arc::DataPoint.

Definition at line 96 of file DataPointDirect.cpp.

                                                          {
    range_start = start;
    range_end = end;
  }
bool Arc::DataPointDirect::ReadOnly ( ) const [virtual, inherited]

Implements Arc::DataPoint.

Definition at line 64 of file DataPointDirect.cpp.

                                       {
    return readonly;
  }
void Arc::DataPointDirect::ReadOutOfOrder ( bool  v) [virtual, inherited]

Allow/disallow DataPoint to produce scattered data during reading operation.

Parameters:
vtrue if allowed (default is false).

Implements Arc::DataPoint.

Definition at line 68 of file DataPointDirect.cpp.

                                               {
    allow_out_of_order = val;
  }
bool Arc::DataPointDirect::Registered ( ) const [virtual, inherited]

Check if file is registered in Indexing Service.

Proper value is obtainable only after Resolve.

Implements Arc::DataPoint.

Definition at line 130 of file DataPointDirect.cpp.

                                         {
    return false;
  }

Remove/delete object at URL.

Implements Arc::DataPoint.

Definition at line 164 of file DataPointGridFTP.cpp.

                                      {
    if (!ftp_active)
      return DataStatus::NotInitializedError;
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    GlobusResult res;
    set_attributes();
    res = globus_ftp_client_delete(&ftp_handle, url.str().c_str(),
                                   &ftp_opattr, &ftp_complete_callback, this);
    if (!res) {
      logger.msg(VERBOSE, "delete_ftp: globus_ftp_client_delete failed");
      logger.msg(INFO, "Globus error: %s", res.str());
      return DataStatus::DeleteError;
    }
    if (!cond.wait(1000*usercfg.Timeout())) {
      logger.msg(INFO, "delete_ftp: globus_ftp_client_delete timeout");
      globus_ftp_client_abort(&ftp_handle);
      cond.wait();
      return DataStatus::DeleteError;
    }
    return condstatus;
  }

Here is the call graph for this function:

Remove current URL from list.

Implements Arc::DataPoint.

Definition at line 165 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::RemoveLocations ( const DataPoint p) [virtual, inherited]

Remove locations present in another DataPoint object.

Implements Arc::DataPoint.

Definition at line 169 of file DataPointDirect.cpp.

DataStatus Arc::DataPointDirect::Resolve ( bool  source) [virtual, inherited]

Resolves index service URL into list of ordinary URLs.

Also obtains meta information about the file.

Parameters:
sourcetrue if DataPoint object represents source of information.

Implements Arc::DataPoint.

Definition at line 102 of file DataPointDirect.cpp.

void Arc::DataPointGridFTP::set_attributes ( void  ) [private]

Definition at line 735 of file DataPointGridFTP.cpp.

                                            {
    globus_ftp_control_parallelism_t paral;
    if (ftp_threads > 1) {
      paral.fixed.mode = GLOBUS_FTP_CONTROL_PARALLELISM_FIXED;
      paral.fixed.size = ftp_threads;
    }
    else {
      paral.fixed.mode = GLOBUS_FTP_CONTROL_PARALLELISM_NONE;
      paral.fixed.size = 1;
    }
    globus_ftp_client_operationattr_set_parallelism(&ftp_opattr, &paral);
    globus_ftp_client_operationattr_set_striped(&ftp_opattr, GLOBUS_FALSE);
    /*   globus_ftp_client_operationattr_set_layout         */
    /*   globus_ftp_client_operationattr_set_tcp_buffer     */
    globus_ftp_client_operationattr_set_type(&ftp_opattr,
                                             GLOBUS_FTP_CONTROL_TYPE_IMAGE);
    if (!is_secure) { // plain ftp protocol
      globus_ftp_client_operationattr_set_mode(&ftp_opattr,
                                               GLOBUS_FTP_CONTROL_MODE_STREAM);
      globus_ftp_client_operationattr_set_data_protection(&ftp_opattr,
                                                          GLOBUS_FTP_CONTROL_PROTECTION_CLEAR);
      globus_ftp_client_operationattr_set_control_protection(&ftp_opattr,
                                                             GLOBUS_FTP_CONTROL_PROTECTION_CLEAR);
      // need to set dcau to none in order Globus libraries not to send
      // it to pure ftp server
      globus_ftp_control_dcau_t dcau;
      dcau.mode = GLOBUS_FTP_CONTROL_DCAU_NONE;
      globus_ftp_client_operationattr_set_dcau(&ftp_opattr, &dcau);
    }
    else { // gridftp protocol

      if (!credential)
        credential = new GSSCredential(usercfg.ProxyPath(),
                                       usercfg.CertificatePath(), usercfg.KeyPath());

      GlobusResult r = globus_ftp_client_operationattr_set_authorization(
                     &ftp_opattr,
                     *credential,":globus-mapping:","user@",
                     GLOBUS_NULL,GLOBUS_NULL);
      if(!r) {
        logger.msg(WARNING, "Failed to set credentials for GridFTP transfer");
        logger.msg(VERBOSE, "globus_ftp_client_operationattr_set_authorization: error: %s", r.str());
      }
      if (force_secure || (url.Option("secure") == "yes")) {
        globus_ftp_client_operationattr_set_mode(&ftp_opattr,
                                                 GLOBUS_FTP_CONTROL_MODE_EXTENDED_BLOCK);
        globus_ftp_client_operationattr_set_data_protection(&ftp_opattr,
                                                            GLOBUS_FTP_CONTROL_PROTECTION_PRIVATE);
        logger.msg(VERBOSE, "Using secure data transfer");
      }
      else {
        if (force_passive)
          globus_ftp_client_operationattr_set_mode(&ftp_opattr,
                                                   GLOBUS_FTP_CONTROL_MODE_STREAM);
        else
          globus_ftp_client_operationattr_set_mode(&ftp_opattr,
                                                   GLOBUS_FTP_CONTROL_MODE_EXTENDED_BLOCK);
        globus_ftp_client_operationattr_set_data_protection(&ftp_opattr,
                                                            GLOBUS_FTP_CONTROL_PROTECTION_CLEAR);
        logger.msg(VERBOSE, "Using insecure data transfer");
      }
      globus_ftp_client_operationattr_set_control_protection(&ftp_opattr,
                                                             GLOBUS_FTP_CONTROL_PROTECTION_PRIVATE);
    }
    /*   globus_ftp_client_operationattr_set_dcau                         */
    /*   globus_ftp_client_operationattr_set_resume_third_party_transfer  */
    /*   globus_ftp_client_operationattr_set_authorization                */
    globus_ftp_client_operationattr_set_append(&ftp_opattr, GLOBUS_FALSE);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointDirect::SetAdditionalChecks ( bool  v) [virtual, inherited]

Allow/disallow additional checks.

Check for existance of remote file (and probably other checks too) before initiating reading and writing operations.

Parameters:
vtrue if allowed (default is true).

Implements Arc::DataPoint.

Definition at line 76 of file DataPointDirect.cpp.

                                                    {
    additional_checks = val;
  }
void Arc::DataPoint::SetCheckSum ( const std::string &  val) [virtual, inherited]

Set value of meta-information 'checksum'.

Definition at line 106 of file DataPoint.cpp.

                                                  {
    checksum = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetCreated ( const Time val) [virtual, inherited]

Set value of meta-information 'creation/modification time'.

Definition at line 122 of file DataPoint.cpp.

                                            {
    created = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetMeta ( const DataPoint p) [virtual, inherited]

Copy meta information from another object.

Already defined values are not overwritten.

Parameters:
pobject from which information is taken.

Definition at line 154 of file DataPoint.cpp.

                                            {
    if (!CheckSize())
      SetSize(p.GetSize());
    if (!CheckCheckSum())
      SetCheckSum(p.GetCheckSum());
    if (!CheckCreated())
      SetCreated(p.GetCreated());
    if (!CheckValid())
      SetValid(p.GetValid());
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void Arc::DataPointDirect::SetSecure ( bool  v) [virtual, inherited]

Allow/disallow heavy security during data transfer.

Parameters:
vtrue if allowed (default depends on protocol).

Implements Arc::DataPoint.

Definition at line 84 of file DataPointDirect.cpp.

                                          {
    force_secure = val;
  }
void Arc::DataPoint::SetSize ( const unsigned long long int  val) [virtual, inherited]

Set value of meta-information 'size'.

Definition at line 94 of file DataPoint.cpp.

                                                          {
    size = val;
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetTries ( const int  n) [virtual, inherited]

Set number of retries.

Reimplemented in Arc::DataPointIndex.

Definition at line 146 of file DataPoint.cpp.

                                      {
    triesleft = std::max(0, n);
  }

Here is the caller graph for this function:

void Arc::DataPoint::SetValid ( const Time val) [virtual, inherited]

Set value of meta-information 'validity time'.

Definition at line 134 of file DataPoint.cpp.

                                          {
    valid = val;
  }

Here is the caller graph for this function:

Start reading data from URL.

Separate thread to transfer data will be created. No other operation can be performed while reading is in progress.

Parameters:
bufferoperation will use this buffer to put information into. Should not be destroyed before stop_reading was called and returned.

Implements Arc::DataPoint.

Definition at line 247 of file DataPointGridFTP.cpp.

                                                           {
    if (!ftp_active)
      return DataStatus::NotInitializedError;
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    set_attributes();
    reading = true;
    buffer = &buf;
    bool limit_length = false;
    unsigned long long int range_length = 0;
    if (range_end > range_start) {
      range_length = range_end - range_start;
      limit_length = true;
    }
    logger.msg(VERBOSE, "start_reading_ftp");
    ftp_eof_flag = false;
    globus_ftp_client_handle_cache_url_state(&ftp_handle, url.str().c_str());
    GlobusResult res;
    logger.msg(VERBOSE, "start_reading_ftp: globus_ftp_client_get");
    if (limit_length)
      res = globus_ftp_client_partial_get(&ftp_handle, url.str().c_str(),
                                          &ftp_opattr, GLOBUS_NULL,
                                          range_start,
                                          range_start + range_length + 1,
                                          &ftp_get_complete_callback, this);
    else
      res = globus_ftp_client_get(&ftp_handle, url.str().c_str(),
                                  &ftp_opattr, GLOBUS_NULL,
                                  &ftp_get_complete_callback, this);
    if (!res) {
      logger.msg(VERBOSE, "start_reading_ftp: globus_ftp_client_get failed");
      logger.msg(INFO, "Globus error: %s", res.str());
      globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
      buffer->error_read(true);
      reading = false;
      return DataStatus::ReadStartError;
    }
    if (globus_thread_create(&ftp_control_thread, GLOBUS_NULL,
                             &ftp_read_thread, this) != 0) {
      logger.msg(VERBOSE, "start_reading_ftp: globus_thread_create failed");
      globus_ftp_client_abort(&ftp_handle);
      cond.wait();
      globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
      buffer->error_read(true);
      reading = false;
      return DataStatus::ReadStartError;
    }
    // make sure globus has thread for handling network/callbacks
    globus_thread_blocking_will_block();
    return DataStatus::Success;
  }

Here is the call graph for this function:

DataStatus Arc::DataPointGridFTP::StartWriting ( DataBuffer buffer,
DataCallback space_cb = NULL 
) [virtual]

Start writing data to URL.

Separate thread to transfer data will be created. No other operation can be performed while writing is in progress.

Parameters:
bufferoperation will use this buffer to get information from. Should not be destroyed before stop_writing was called and returned.
space_cbcallback which is called if there is not enough space to store data. May not implemented for all protocols.

Implements Arc::DataPoint.

Definition at line 408 of file DataPointGridFTP.cpp.

                                                           {
    if (!ftp_active)
      return DataStatus::NotInitializedError;
    if (reading)
      return DataStatus::IsReadingError;
    if (writing)
      return DataStatus::IsWritingError;
    set_attributes();
    writing = true;
    buffer = &buf;
    /* size of file first */
    bool limit_length = false;
    unsigned long long int range_length = 0;
    if (range_end > range_start) {
      range_length = range_end - range_start;
      limit_length = true;
    }
    ftp_eof_flag = false;
    GlobusResult res;
    globus_ftp_client_handle_cache_url_state(&ftp_handle, url.str().c_str());
    if (autodir) {
      logger.msg(VERBOSE, "start_writing_ftp: mkdir");
      if (!mkdir_ftp())
        logger.msg(VERBOSE,
                   "start_writing_ftp: mkdir failed - still trying to write");
    }
    logger.msg(VERBOSE, "start_writing_ftp: put");
    if (limit_length)
      res = globus_ftp_client_partial_put(&ftp_handle, url.str().c_str(),
                                          &ftp_opattr, GLOBUS_NULL,
                                          range_start,
                                          range_start + range_length,
                                          &ftp_put_complete_callback, this);
    else
      res = globus_ftp_client_put(&ftp_handle, url.str().c_str(),
                                  &ftp_opattr, GLOBUS_NULL,
                                  &ftp_put_complete_callback, this);
    if (!res) {
      logger.msg(VERBOSE, "start_writing_ftp: put failed");
      logger.msg(INFO, "Globus error: %s", res.str());
      globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
      buffer->error_write(true);
      writing = false;
      return DataStatus::WriteStartError;
    }
    if (globus_thread_create(&ftp_control_thread, GLOBUS_NULL,
                             &ftp_write_thread, this) != 0) {
      logger.msg(VERBOSE, "start_writing_ftp: globus_thread_create failed");
      globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
      buffer->error_write(true);
      writing = false;
      return DataStatus::WriteStartError;
    }
    // make sure globus has thread for handling network/callbacks
    globus_thread_blocking_will_block();
    return DataStatus::Success;
  }

Here is the call graph for this function:

Stop reading.

Must be called after corresponding start_reading method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred. Must return failure if any happened during transfer.

Implements Arc::DataPoint.

Definition at line 301 of file DataPointGridFTP.cpp.

                                           {
    if (!reading)
      return DataStatus::ReadStopError;
    reading = false;
    if (!buffer->eof_read()) {
      logger.msg(VERBOSE, "stop_reading_ftp: aborting connection");
      globus_ftp_client_abort(&ftp_handle);
    }
    logger.msg(VERBOSE, "stop_reading_ftp: waiting for transfer to finish");
    cond.wait();
    logger.msg(VERBOSE, "stop_reading_ftp: exiting: %s", url.str());
    globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
    return condstatus;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Stop writing.

Must be called after corresponding start_writing method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred. Must return failure if any happened during transfer.

Implements Arc::DataPoint.

Definition at line 467 of file DataPointGridFTP.cpp.

                                           {
    if (!writing)
      return DataStatus::WriteStopError;
    writing = false;
    if (!buffer->eof_write()) {
      logger.msg(VERBOSE, "StopWriting: aborting connection");
      globus_ftp_client_abort(&ftp_handle);
    }
    cond.wait();
    globus_ftp_client_handle_flush_url_state(&ftp_handle, url.str().c_str());
    return condstatus;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

std::string Arc::DataPoint::str ( ) const [virtual, inherited]

Returns a string representation of the DataPoint.

Definition at line 53 of file DataPoint.cpp.

                                 {
    return url.str();
  }

Here is the call graph for this function:

Here is the caller graph for this function:

DataStatus Arc::DataPointDirect::Unregister ( bool  all) [virtual, inherited]

Index Service unregistration.

Remove information about file registered in Indexing Service.

Parameters:
allif true, information about file itself is (LFN) is removed. Otherwise only particular physical instance is unregistered.

Implements Arc::DataPoint.

Definition at line 118 of file DataPointDirect.cpp.

Returns true if URL can accept scattered data for writing operation.

Reimplemented from Arc::DataPointDirect.

Definition at line 839 of file DataPointGridFTP.cpp.

                                         {
    return true;
  }

Member Data Documentation

bool Arc::DataPointDirect::additional_checks [protected, inherited]

Definition at line 84 of file DataPointDirect.h.

bool Arc::DataPointDirect::allow_out_of_order [protected, inherited]

Definition at line 85 of file DataPointDirect.h.

Definition at line 29 of file DataPointGridFTP.h.

DataBuffer* Arc::DataPointDirect::buffer [protected, inherited]

Definition at line 75 of file DataPointDirect.h.

int Arc::DataPointDirect::bufnum [protected, inherited]

Definition at line 77 of file DataPointDirect.h.

long long int Arc::DataPointDirect::bufsize [protected, inherited]

Definition at line 76 of file DataPointDirect.h.

bool Arc::DataPoint::cache [protected, inherited]

Definition at line 325 of file DataPoint.h.

std::string Arc::DataPoint::checksum [protected, inherited]

Definition at line 320 of file DataPoint.h.

Definition at line 31 of file DataPointGridFTP.h.

Definition at line 32 of file DataPointGridFTP.h.

Time Arc::DataPoint::created [protected, inherited]

Definition at line 321 of file DataPoint.h.

Definition at line 33 of file DataPointGridFTP.h.

DataStatus Arc::DataPoint::failure_code [protected, inherited]

Definition at line 324 of file DataPoint.h.

bool Arc::DataPointDirect::force_passive [protected, inherited]

Definition at line 83 of file DataPointDirect.h.

bool Arc::DataPointDirect::force_secure [protected, inherited]

Definition at line 82 of file DataPointDirect.h.

Definition at line 24 of file DataPointGridFTP.h.

char Arc::DataPointGridFTP::ftp_buf[16] [private]

Definition at line 69 of file DataPointGridFTP.h.

globus_thread_t Arc::DataPointGridFTP::ftp_control_thread [private]

Definition at line 27 of file DataPointGridFTP.h.

std::string Arc::DataPointGridFTP::ftp_dir_path [private]

Definition at line 67 of file DataPointGridFTP.h.

Definition at line 38 of file DataPointGridFTP.h.

globus_ftp_client_handle_t Arc::DataPointGridFTP::ftp_handle [private]

Definition at line 25 of file DataPointGridFTP.h.

globus_ftp_client_operationattr_t Arc::DataPointGridFTP::ftp_opattr [private]

Definition at line 26 of file DataPointGridFTP.h.

Definition at line 28 of file DataPointGridFTP.h.

bool Arc::DataPointDirect::is_secure [protected, inherited]

Definition at line 81 of file DataPointDirect.h.

bool Arc::DataPointDirect::linkable [protected, inherited]

Definition at line 80 of file DataPointDirect.h.

bool Arc::DataPointDirect::local [protected, inherited]

Definition at line 78 of file DataPointDirect.h.

Reimplemented from Arc::DataPoint.

Definition at line 23 of file DataPointGridFTP.h.

unsigned long long int Arc::DataPointDirect::range_end [protected, inherited]

Definition at line 87 of file DataPointDirect.h.

unsigned long long int Arc::DataPointDirect::range_start [protected, inherited]

Definition at line 86 of file DataPointDirect.h.

Definition at line 35 of file DataPointGridFTP.h.

bool Arc::DataPointDirect::readonly [protected, inherited]

Definition at line 79 of file DataPointDirect.h.

unsigned long long int Arc::DataPoint::size [protected, inherited]

Definition at line 319 of file DataPoint.h.

int Arc::DataPoint::triesleft [protected, inherited]

Definition at line 323 of file DataPoint.h.

const URL& Arc::DataPoint::url [protected, inherited]

Definition at line 315 of file DataPoint.h.

const UserConfig& Arc::DataPoint::usercfg [protected, inherited]

Definition at line 316 of file DataPoint.h.

Time Arc::DataPoint::valid [protected, inherited]

Definition at line 322 of file DataPoint.h.

std::list<std::string> Arc::DataPoint::valid_url_options [protected, inherited]

Subclasses should add their own specific options to this list.

Definition at line 327 of file DataPoint.h.

Definition at line 36 of file DataPointGridFTP.h.


The documentation for this class was generated from the following files: