Back to index

nordugrid-arc-nox  1.1.0~rc6
Public Member Functions | Public Attributes | Private Member Functions | Private Attributes
Arc::JobLogFile Class Reference

Class to represent a job log file created by A-REX, and to create OGF Job Usage Records from them. More...

#include <JobLogFile.h>

Inheritance diagram for Arc::JobLogFile:
Inheritance graph
[legend]
Collaboration diagram for Arc::JobLogFile:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 JobLogFile (const std::string &_filename)
 Constructor.
int parse (const std::string &_filename)
 Reloads and parses A-REX job log.
void createUsageRecord (Arc::XMLNode &usagerecord, const char *recordid_prefix="ur-")
 Creates an OGF Job Usage Record from parsed log files.
std::string getFilename ()
 Returns original full path to log file.
void allowRemove (bool a)
 Enables/disables file removal from disk.
bool exists ()
 Checks if file exists on the disk.
bool olderThan (time_t age)
 Checks if file was modified earlier than 'age' seconds ago.
void remove ()
 Deletes file from the disk.

Public Attributes

keys
 STL member.
T elements
 STL member.

Private Member Functions

std::string getArchivingPath ()

Private Attributes

std::string filename
bool allow_remove

Detailed Description

Class to represent a job log file created by A-REX, and to create OGF Job Usage Records from them.

Definition at line 15 of file JobLogFile.h.


Constructor & Destructor Documentation

Arc::JobLogFile::JobLogFile ( const std::string &  _filename) [inline]

Constructor.

Loads and parses A-REX job log.

Definition at line 26 of file JobLogFile.h.

:allow_remove(true) { parse(_filename); } 

Here is the call graph for this function:


Member Function Documentation

void Arc::JobLogFile::allowRemove ( bool  a) [inline]

Enables/disables file removal from disk.

Definition at line 41 of file JobLogFile.h.

{ allow_remove=a; }

Here is the caller graph for this function:

void Arc::JobLogFile::createUsageRecord ( Arc::XMLNode usagerecord,
const char *  recordid_prefix = "ur-" 
)

Creates an OGF Job Usage Record from parsed log files.

  • Missing UR properties:
    1. ProcessID: Local PID(s) of job. Extraction is LRMS-specific and
      may not always be possible
    2. Charge: Amount of money or abstract credits charged for the job.
    3. Some differentiated properties e.g. network, disk etc.

Definition at line 82 of file JobLogFile.cpp.

  {
    //***
    //If archiving is enabled: first try to load archived UR
    std::string archive_fn=getArchivingPath();
    if (!archive_fn.empty())
      {
       errno=0;
       if (usagerecord.ReadFromFile(archive_fn))
         {
           Arc::Logger::rootLogger.msg(Arc::VERBOSE,
              "Read archive file %s",
              archive_fn.c_str());
           return;
         }
       Arc::Logger::rootLogger.msg(Arc::VERBOSE,
          "Could not read archive file %s for job log file %s (%s), generating new UR",
          archive_fn.c_str(),
          filename.c_str(),
          strerror(errno));
      }
    //Otherwise go on and create new UR
    //***

    Arc::NS ns_ur;
    
    //Namespaces defined by OGF
    ns_ur[""]="http://schema.ogf.org/urf/2003/09/urf";
    ns_ur["urf"]="http://schema.ogf.org/urf/2003/09/urf";
    ns_ur["xsd"]="http://www.w3.org/2001/XMLSchema";
    ns_ur["xsi"]="http://www.w3.org/2001/XMLSchema-instance";
    ns_ur["ds"]="http://www.w3.org/2000/09/xmldsig#";
    ns_ur["arc"]="http://www.nordugrid.org/ws/schemas/ur-arc";

    //Get node names
    std::list<std::string> nodenames;
    std::string mainnode;
    std::string nodestr=(*this)["nodename"];
    size_type pcolon=nodestr.find(':');
    while (pcolon!=std::string::npos)
      {
       nodenames.push_back(nodestr.substr(0,pcolon));
       nodestr=nodestr.substr(pcolon+1,std::string::npos);
       pcolon=nodestr.find(':');
      }
    if (!nodestr.empty()) nodenames.push_back(nodestr);
    if (!nodenames.empty()) mainnode=*(nodenames.begin());

    //Get runtime environments
    std::list<std::string> rtes;
    std::string rtestr=(*this)["runtimeenvironment"];
    size_type pspace=rtestr.find(" ");
    while (pspace!=std::string::npos)
      {
       std::string rte=rtestr.substr(0,pspace);
       if (!rte.empty()) rtes.push_back(rte);
       rtestr=rtestr.substr(pspace+1,std::string::npos);
       pspace=rtestr.find(" ");
      }
    if (!rtestr.empty()) rtes.push_back(rtestr);
    
    //Fill this Usage Record
    Arc::XMLNode ur(ns_ur,"JobUsageRecord");
    
    //RecordIdentity, GlobalJobId, LocalJobId
    if (find("ngjobid")!=end())
      {
       // Timestamp for record, required
       std::string nowstamp=Arc::Time().str(Arc::UTCTime);
       
       Arc::XMLNode rid=ur.NewChild("RecordIdentity");
       rid.NewAttribute("createTime")=nowstamp;
       //NOTE! Current LUTS also sets a "creationTime"[sic!] for each record
       
       // ID for record
       if (!mainnode.empty())
         rid.NewAttribute("recordId")=
           std::string(recordid_prefix) + mainnode + 
           '-' + (*this)["ngjobid"];
       else
         rid.NewAttribute("recordId")=
           std::string(recordid_prefix) + (*this)["ngjobid"];
       
       ur.NewChild("JobIdentity").NewChild("GlobalJobId")=
         (*this)["globalid"];
       
       if (find("localjobid")!=end())
         ur["JobIdentity"].NewChild("LocalJobId")=
           (*this)["localjobid"];
      }
    else
    {
      //TODO what if not valid?
      Arc::Logger::rootLogger.msg(Arc::VERBOSE,
                    "Missing required UR element \"RecordIdentity\", in job log file %s",
                    filename.c_str());
      usagerecord.Destroy();
      return;
    }
    
    //ProcessId?
    
    //GlobalUser[Nn]ame, LocalUserId
    //TODO clarify case
    //NOTE! original JARM used "GlobalUserId"
    if (find("usersn")!=end())
      {
       ur.NewChild("UserIdentity").NewChild("GlobalUserName")=
         (*this)["usersn"];
       
       if (find("localuser")!=end())
         ur["UserIdentity"].NewChild("LocalUserId")=
           (*this)["localuser"];
      }
    
    //JobName
    if (find("jobname")!=end())
      {
       ur.NewChild("JobName")=(*this)["jobname"];
      }
    
    //Charge?
    
    //Status
    if (find("status")!=end())
      {
       ur.NewChild("Status")=(*this)["status"];  //TODO convert?
      }
    else
      {
       //TODO what if not valid?
       Arc::Logger::rootLogger.msg(Arc::VERBOSE,
                     "Missing required element \"Status\" in job log file %s",
                     filename.c_str());
       usagerecord.Destroy();
       return;
      }
    
    //---
    //Network?

    //Disk?

    //Memory
    if (find("usedmemory")!=end())
      {
       Arc::XMLNode memn=ur.NewChild("Memory")=(*this)["usedmemory"];
       memn.NewAttribute("storageUnit")="kB";
       memn.NewAttribute("metric")="average";
       memn.NewAttribute("type")="virtual";
      }

    if (find("usedmaxresident")!=end())
      {
       Arc::XMLNode memn=ur.NewChild("Memory")=(*this)["usedmaxresident"];
       memn.NewAttribute("storageUnit")="kB";
       memn.NewAttribute("metric")="max";
       memn.NewAttribute("type")="physical";
      }

    if (find("usedaverageresident")!=end())
      {
       Arc::XMLNode memn=ur.NewChild("Memory")=(*this)["usedaverageresident"];
       memn.NewAttribute("storageUnit")="kB";
       memn.NewAttribute("metric")="average";
       memn.NewAttribute("type")="physical";
      }
    
    //Swap?

    //TimeDuration, TimeInstant, ServiceLevel?

    //---
    //WallDuration
    if (find("usedwalltime")!=end())
      {
       Arc::Period walldur((*this)["usedwalltime"],Arc::PeriodSeconds);
       ur.NewChild("WallDuration")=(std::string)walldur;
      }
    
    //CpuDuration

    if (find("usedusercputime")!=end() && find("usedkernelcputime")!=end())
      {
       Arc::Period udur((*this)["usedusercputime"],Arc::PeriodSeconds);
       Arc::Period kdur((*this)["usedkernelcputime"],Arc::PeriodSeconds);

       Arc::XMLNode udurn=ur.NewChild("CpuDuration")=(std::string)udur;
       udurn.NewAttribute("usageType")="user";

       Arc::XMLNode kdurn=ur.NewChild("CpuDuration")=(std::string)kdur;
       kdurn.NewAttribute("usageType")="kernel";
      }
    else
    if (find("usedcputime")!=end())
      {
       Arc::Period cpudur((*this)["usedcputime"],Arc::PeriodSeconds);
       ur.NewChild("CpuDuration")=(std::string)cpudur;
      }
    
    //StartTime
    if (find("submissiontime")!=end())
      {
       Arc::Time starttime((*this)["submissiontime"]);
       ur.NewChild("StartTime")=starttime.str(Arc::UTCTime);
      }
    
    //EndTime
    if (find("endtime")!=end())
      {
       Arc::Time endtime((*this)["endtime"]);
       ur.NewChild("EndTime")=endtime.str(Arc::UTCTime);
      }
    
    //MachineName
    if (find("nodename")!=end())
      {
       ur.NewChild("MachineName")=mainnode;
      }
    
    //Host
    if (!mainnode.empty())
      {
       Arc::XMLNode primary_node=ur.NewChild("Host");
       primary_node=mainnode;
       primary_node.NewAttribute("primary")="true";
       std::list<std::string>::iterator it=nodenames.begin();
       ++it;
       while (it!=nodenames.end())
         {
           ur.NewChild("Host")=*it;
           ++it;
         }    
      }
    
    //SubmitHost
    if (find("clienthost")!=end())
      {
       // Chop port no.
       std::string hostport=(*this)["clienthost"], host;
       size_type clnp=hostport.find(":");
       if (clnp==std::string::npos)
         host=hostport;
       else
         host=hostport.substr(0,clnp);

       ur.NewChild("SubmitHost")=host;
      }
    
    //Queue
    if (find("lrms")!=end())
      {
       ur.NewChild("Queue")=(*this)["lrms"];  //OK for Queue?
      }
    
    //ProjectName
    if (find("projectname")!=end())
      {
       ur.NewChild("ProjectName")=(*this)["projectname"];
      }

    
    //NodeCount
    if (find("nodecount")!=end())
      {
       ur.NewChild("NodeCount")=(*this)["nodecount"];
      }

    //Processors?

    //Extra:
    //RunTimeEnvironment

    for(std::list<std::string>::iterator jt=rtes.begin();
       jt!=rtes.end();
       ++jt)
      {
       ur.NewChild("arc:RunTimeEnvironment")=*jt;
      }
    
    //TODO user id info

    //***
    //Archiving if enabled:
    if (!archive_fn.empty())
      {
       struct stat st;
       std::string dir_name=(*this)["jobreport_option_archiving"];
       if (stat(dir_name.c_str(),&st)!=0)
         {
           Arc::Logger::rootLogger.msg(Arc::VERBOSE,
                                    "Creating directory %s",
                                    dir_name.c_str());
           errno=0;
           if (mkdir(dir_name.c_str(),S_IRWXU)!=0)
             {
              Arc::Logger::rootLogger.msg(Arc::ERROR,
                  "Failed to create archive directory %s: %s",
                  dir_name.c_str(),
                  strerror(errno));
             }
         }
       
       Arc::Logger::rootLogger.msg(Arc::VERBOSE,
                                "Archiving UR to file %s",
                                archive_fn.c_str());
       errno=0;
       if (!ur.SaveToFile(archive_fn.c_str()))
         {
           Arc::Logger::rootLogger.msg(Arc::ERROR,
                                   "Failed to write file %s: %s",
                                   archive_fn.c_str(),
                                   strerror(errno));
         }
      }
    //***

    usagerecord.Replace(ur);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Checks if file exists on the disk.

Definition at line 403 of file JobLogFile.cpp.

  {
    //TODO cross-platform?
    struct stat s;
    return (0==stat(filename.c_str(),&s));
  }

Here is the caller graph for this function:

std::string Arc::JobLogFile::getArchivingPath ( ) [private]

Definition at line 429 of file JobLogFile.cpp.

  {
    //no archiving dir set
    if ((*this)["jobreport_option_archiving"].empty()) return std::string();

    //if set, archive file name corresponds to original job log file
    std::string base_fn;
    size_type seppos=filename.rfind('/');
    if (seppos==std::string::npos)
      base_fn=filename;
    else
      base_fn=filename.substr(seppos+1,std::string::npos);

    return (*this)["jobreport_option_archiving"]+"/usagerecord."+base_fn;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

std::string Arc::JobLogFile::getFilename ( ) [inline]

Returns original full path to log file.

Definition at line 39 of file JobLogFile.h.

{ return filename; }

Here is the caller graph for this function:

bool Arc::JobLogFile::olderThan ( time_t  age)

Checks if file was modified earlier than 'age' seconds ago.

Definition at line 410 of file JobLogFile.cpp.

  {
    struct stat s;
    return ( ( 0==stat(filename.c_str(),&s) ) &&
            ( (time(NULL)-s.st_mtime) > age ) 
            );
  }

Here is the caller graph for this function:

int Arc::JobLogFile::parse ( const std::string &  _filename)

Reloads and parses A-REX job log.

Definition at line 21 of file JobLogFile.cpp.

  {
    int count=0;  //number of parsed values
    clear();
    filename=_filename;
    if (!exists()) return -1;

    std::ifstream logfile(filename.c_str(),std::ios::in);
    std::string line;
    while (logfile.good())
      {
       std::getline(logfile,line);
       size_type e=line.find('=');
       if (e!=std::string::npos)
         {
           count++;
           std::string key=line.substr(0, e),
             value=line.substr(e+1, std::string::npos);
           (*this)[key]=value;
         }
      }
    logfile.close();

    //Parse jobreport_options string!

    std::string jobreport_opts=(*this)["accounting_options"];
    std::string option;
    size_type pcomma=jobreport_opts.find(',');
    size_type pcolon;
    while (pcomma!=std::string::npos)
      {
       option=jobreport_opts.substr(0,pcomma);
       // separate opt_name:value pair
       pcolon=option.find(':');
       if (pcolon!=std::string::npos)
         {
           std::string key=option.substr(0, pcolon), 
             value=option.substr(pcolon+1, std::string::npos);

           (*this)[std::string("jobreport_option_")+key]=value;
         }
       
       //next:
       jobreport_opts=jobreport_opts.substr(pcomma+1, std::string::npos);
       pcomma=jobreport_opts.find(',');
      }
    option=jobreport_opts;
    pcolon=option.find(':');
    if (pcolon!=std::string::npos)
      {
       std::string key=option.substr(0, pcolon), 
         value=option.substr(pcolon+1, std::string::npos);
       
       (*this)[std::string("jobreport_option_")+key]=value;
      }



    return count;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Deletes file from the disk.

Definition at line 418 of file JobLogFile.cpp.

  {
    if (!allow_remove) return;
    errno=0;
    int e = ::remove(filename.c_str());
    if (e)
      Arc::Logger::rootLogger.msg(Arc::ERROR,"Failed to delete file %s:%s",
                    filename.c_str(),
                    strerror(errno));
  }

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Definition at line 22 of file JobLogFile.h.

template<typename K, typename T>
T std::map< K, T >::elements [inherited]

STL member.

std::string Arc::JobLogFile::filename [private]

Definition at line 21 of file JobLogFile.h.

template<typename K, typename T>
K std::map< K, T >::keys [inherited]

STL member.


The documentation for this class was generated from the following files: