Back to index

lightning-sunbird  0.9+nobinonly
Public Member Functions | Static Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes
nsScanner Class Reference

#include <nsScanner.h>

Collaboration diagram for nsScanner:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 nsScanner (const nsAString &anHTMLString, const nsACString &aCharset, PRInt32 aSource)
 Use this constructor if you want i/o to be based on a single string you hand in during construction.
 nsScanner (nsString &aFilename, PRBool aCreateStream, const nsACString &aCharset, PRInt32 aSource)
 Use this constructor if you want i/o to be based on a file (therefore a stream) or just data you provide via Append().
 nsScanner (const nsAString &aFilename, nsIInputStream *aStream, const nsACString &aCharset, PRInt32 aSource)
 Use this constructor if you want i/o to be stream based.
 ~nsScanner ()
 default destructor
nsresult GetChar (PRUnichar &ch)
 retrieve next char from internal input stream
nsresult Peek (PRUnichar &ch, PRUint32 aOffset=0)
 peek ahead to consume next char from scanner's internal input buffer
nsresult Peek (nsAString &aStr, PRInt32 aNumChars, PRInt32 aOffset=0)
nsresult SkipOver (nsString &SkipChars)
 Skip over chars as long as they're in aSkipSet.
nsresult SkipOver (PRUnichar aSkipChar)
 Skip over chars as long as they equal given char.
nsresult SkipTo (nsString &aValidSet)
 Skip over chars until they're in aValidSet.
nsresult SkipPast (nsString &aSequence)
 Skip over chars as long as they're in aSequence.
nsresult SkipWhitespace (PRInt32 &aNewlinesSkipped)
 Skip whitespace on scanner input stream.
nsresult ReadTagIdentifier (nsScannerSharedSubstring &aString)
 Consume characters until you run into space, a '<', a '>', or a '/'.
nsresult ReadEntityIdentifier (nsString &aString)
 Consume characters until you run into a char that's not valid in an entity name.
nsresult ReadNumber (nsString &aString, PRInt32 aBase)
 Consume digits.
nsresult ReadWhitespace (nsScannerSharedSubstring &aString, PRInt32 &aNewlinesSkipped, PRBool &aHaveCR)
 Consume characters until you find the terminal char.
nsresult ReadWhitespace (nsScannerIterator &aStart, nsScannerIterator &aEnd, PRInt32 &aNewlinesSkipped)
nsresult ReadUntil (nsAString &aString, PRUnichar aTerminal, PRBool addTerminal)
 Consume characters until you find the terminal char.
nsresult ReadUntil (nsAString &aString, const nsReadEndCondition &aEndCondition, PRBool addTerminal)
 Consume characters until you find one contained in given terminal set.
nsresult ReadUntil (nsScannerSharedSubstring &aString, const nsReadEndCondition &aEndCondition, PRBool addTerminal)
nsresult ReadUntil (nsScannerIterator &aStart, nsScannerIterator &aEnd, const nsReadEndCondition &aEndCondition, PRBool addTerminal)
nsresult ReadWhile (nsString &aString, nsString &anInputSet, PRBool addTerminal)
 Consume characters while they're members of anInputSet.
void Mark (void)
 Records current offset position in input stream.
void RewindToMark (void)
 Resets current offset position of input stream to marked position.
PRBool UngetReadable (const nsAString &aBuffer)
 harishd 01/12/99
nsresult Append (const nsAString &aBuffer)
 gess 5/13/98
nsresult Append (const char *aBuffer, PRUint32 aLen, nsIRequest *aRequest)
 gess 5/21/98
void CopyUnusedData (nsString &aCopyBuffer)
 Call this to copy bytes out of the scanner that have not yet been consumed by the tokenization process.
nsStringGetFilename (void)
 Retrieve the name of the file that the scanner is reading from.
nsresult SetDocumentCharset (const nsACString &aCharset, PRInt32 aSource)
 Use this setter to change the scanner's unicode decoder.
void BindSubstring (nsScannerSubstring &aSubstring, const nsScannerIterator &aStart, const nsScannerIterator &aEnd)
void CurrentPosition (nsScannerIterator &aPosition)
void EndReading (nsScannerIterator &aPosition)
void SetPosition (nsScannerIterator &aPosition, PRBool aTruncate=PR_FALSE, PRBool aReverse=PR_FALSE)
void ReplaceCharacter (nsScannerIterator &aPosition, PRUnichar aChar)
PRBool IsIncremental (void)
 Internal method used to cause the internal buffer to be filled with data.
void SetIncremental (PRBool anIncrValue)
PRInt32 FirstNonWhitespacePosition ()
 Return the position of the first non-whitespace character.
void SetParser (nsParser *aParser)
nsresult FillBuffer (void)
 Fill internal buffer with new data, returns an error if no new data is read.

Static Public Member Functions

static void SelfTest ()
 Conduct self test.

Protected Types

enum  { eBufferSizeThreshold = 0x1000 }

Protected Member Functions

void AppendToBuffer (nsScannerString::Buffer *, nsIRequest *aRequest)
void AppendToBuffer (const nsAString &aStr)
void AppendASCIItoBuffer (const char *aData, PRUint32 aLen, nsIRequest *aRequest)

Protected Attributes

nsCOMPtr< nsIInputStreammInputStream
nsScannerStringmSlidingBuffer
nsScannerIterator mCurrentPosition
nsScannerIterator mMarkPosition
nsScannerIterator mEndPosition
nsString mFilename
PRUint32 mCountRemaining
PRUint32 mTotalRead
PRPackedBool mIncremental
PRInt32 mFirstNonWhitespacePosition
PRInt32 mCharsetSource
nsCString mCharset
nsIUnicodeDecodermUnicodeDecoder
nsParsermParser

Detailed Description

Definition at line 74 of file nsScanner.h.


Member Enumeration Documentation

anonymous enum [protected]
Enumerator:
eBufferSizeThreshold 

Definition at line 381 of file nsScanner.h.

{eBufferSizeThreshold=0x1000};  //4K

Constructor & Destructor Documentation

nsScanner::nsScanner ( const nsAString &  anHTMLString,
const nsACString &  aCharset,
PRInt32  aSource 
)

Use this constructor if you want i/o to be based on a single string you hand in during construction.

This short cut was added for Javascript.

ftang 3/02/99

Parameters:
aCharsetcharset
aCharsetSource- where the charset info came from
aModerepresents the parser mode (nav, other)
Returns:

This short cut was added for Javascript.

gess 5/12/98

Parameters:
aModerepresents the parser mode (nav, other)
Returns:

Definition at line 97 of file nsScanner.cpp.

nsScanner::nsScanner ( nsString aFilename,
PRBool  aCreateStream,
const nsACString &  aCharset,
PRInt32  aSource 
)

Use this constructor if you want i/o to be based on a file (therefore a stream) or just data you provide via Append().

Use this constructor if you want i/o to be based on strings the scanner receives.

ftang 3/02/99

Parameters:
aCharsetcharset
aCharsetSource- where the charset info came from
aModerepresents the parser mode (nav, other)
Returns:

If you pass a null filename, you can still provide data to the scanner via append.

gess 5/12/98

Parameters:
aFilename--
Returns:

Definition at line 125 of file nsScanner.cpp.

  : mFilename(aFilename), mParser(nsnull)
{
  MOZ_COUNT_CTOR(nsScanner);

  mSlidingBuffer = nsnull;

  // XXX This is a big hack.  We need to initialize the iterators to something.
  // What matters is that mCurrentPosition == mEndPosition, so that our methods
  // believe that we are at EOF (see bug 182067).  We null out mCurrentPosition
  // so that we have some hope of catching null pointer dereferences associated
  // with this hack. --darin
  memset(&mCurrentPosition, 0, sizeof(mCurrentPosition));
  mMarkPosition = mCurrentPosition;
  mEndPosition = mCurrentPosition;

  mIncremental = PR_TRUE;
  mFirstNonWhitespacePosition = -1;
  mCountRemaining = 0;
  mTotalRead=0;

  if(aCreateStream) {
    nsCOMPtr<nsILocalFile> file;
    nsCOMPtr<nsIInputStream> fileStream;
    
    NS_NewLocalFile(aFilename, PR_TRUE, getter_AddRefs(file));
    if (file)
      NS_NewLocalFileInputStream(getter_AddRefs(mInputStream), file);

  } //if
  mUnicodeDecoder = 0;
  mCharsetSource = kCharsetUninitialized;
  SetDocumentCharset(aCharset, aSource);
}

Here is the call graph for this function:

nsScanner::nsScanner ( const nsAString &  aFilename,
nsIInputStream aStream,
const nsACString &  aCharset,
PRInt32  aSource 
)

Use this constructor if you want i/o to be stream based.

ftang 3/02/99

Parameters:
aCharsetcharset
aCharsetSource- where the charset info came from
aModerepresents the parser mode (nav, other)
Returns:

gess 5/12/98

Parameters:
aStream--
assumeOwnership--
aFilename--
Returns:

Definition at line 170 of file nsScanner.cpp.

  : mFilename(aFilename), mParser(nsnull)
{  
  MOZ_COUNT_CTOR(nsScanner);

  mSlidingBuffer = nsnull;

  // XXX This is a big hack.  We need to initialize the iterators to something.
  // What matters is that mCurrentPosition == mEndPosition, so that our methods
  // believe that we are at EOF (see bug 182067).  We null out mCurrentPosition
  // so that we have some hope of catching null pointer dereferences associated
  // with this hack. --darin
  memset(&mCurrentPosition, 0, sizeof(mCurrentPosition));
  mMarkPosition = mCurrentPosition;
  mEndPosition = mCurrentPosition;

  mIncremental = PR_FALSE;
  mFirstNonWhitespacePosition = -1;
  mCountRemaining = 0;
  mTotalRead=0;
  mInputStream=aStream;
  mUnicodeDecoder = 0;
  mCharsetSource = kCharsetUninitialized;
  SetDocumentCharset(aCharset, aSource);
}

Here is the call graph for this function:

default destructor

gess 3/25/98

Parameters:
@return

Definition at line 252 of file nsScanner.cpp.


Member Function Documentation

nsresult nsScanner::Append ( const nsAString &  aBuffer)

gess 5/13/98

Append data to our underlying input buffer as if it were read from an input stream.

Parameters:
@return

gess4/3/98

Returns:
error code

Definition at line 333 of file nsScanner.cpp.

                                                   {
  mTotalRead += aBuffer.Length();
  AppendToBuffer(aBuffer);
  return NS_OK;
}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::Append ( const char *  aBuffer,
PRUint32  aLen,
nsIRequest aRequest 
)

gess 5/21/98

Parameters:
@return

Definition at line 346 of file nsScanner.cpp.

{
  nsresult res=NS_OK;
  PRUnichar *unichars, *start;
  if(mUnicodeDecoder) {
    PRInt32 unicharBufLen = 0;
    mUnicodeDecoder->GetMaxLength(aBuffer, aLen, &unicharBufLen);
    nsScannerString::Buffer* buffer = nsScannerString::AllocBuffer(unicharBufLen + 1);
    NS_ENSURE_TRUE(buffer,NS_ERROR_OUT_OF_MEMORY);
    start = unichars = buffer->DataStart();

    PRInt32 totalChars = 0;
    PRInt32 unicharLength = unicharBufLen;
    do {
      PRInt32 srcLength = aLen;
      res = mUnicodeDecoder->Convert(aBuffer, &srcLength, unichars, &unicharLength);

      totalChars += unicharLength;
      // Continuation of failure case
      if(NS_FAILED(res)) {
        // if we failed, we consume one byte, replace it with U+FFFD
        // and try the conversion again.

        // This is only needed because some decoders don't follow the
        // nsIUnicodeDecoder contract: they return a failure when *aDestLength
        // is 0 rather than the correct NS_OK_UDEC_MOREOUTPUT.  See bug 244177
        if ((unichars + unicharLength) >= buffer->DataEnd()) {
          NS_ERROR("Unexpected end of destination buffer");
          break;
        }

        unichars[unicharLength++] = (PRUnichar)0xFFFD;
        unichars = unichars + unicharLength;
        unicharLength = unicharBufLen - (++totalChars);

        mUnicodeDecoder->Reset();

        if(((PRUint32) (srcLength + 1)) > aLen) {
          srcLength = aLen;
        }
        else {
          ++srcLength;
        }

        aBuffer += srcLength;
        aLen -= srcLength;
      }
    } while (NS_FAILED(res) && (aLen > 0));

    buffer->SetDataLength(totalChars);
    AppendToBuffer(buffer, aRequest);
    mTotalRead += totalChars;

    // Don't propagate return code of unicode decoder
    // since it doesn't reflect on our success or failure
    // - Ref. bug 87110
    res = NS_OK; 
  }
  else {
    AppendASCIItoBuffer(aBuffer, aLen, aRequest);
    mTotalRead+=aLen;
  }

  return res;
}

Here is the call graph for this function:

void nsScanner::AppendASCIItoBuffer ( const char *  aData,
PRUint32  aLen,
nsIRequest aRequest 
) [protected]

Definition at line 1430 of file nsScanner.cpp.

{
  nsScannerString::Buffer* buf = nsScannerString::AllocBuffer(aLen);
  if (buf)
  {
    LossyConvertEncoding<char, PRUnichar> converter(buf->DataStart());
    converter.write(aData, aLen);
    converter.write_terminator();
    AppendToBuffer(buf, aRequest);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void nsScanner::AppendToBuffer ( nsScannerString::Buffer aBuf,
nsIRequest aRequest 
) [protected]

Definition at line 1387 of file nsScanner.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

void nsScanner::AppendToBuffer ( const nsAString &  aStr) [inline, protected]

Definition at line 384 of file nsScanner.h.

Here is the call graph for this function:

Definition at line 1337 of file nsScanner.cpp.

{
  aSubstring.Rebind(*mSlidingBuffer, aStart, aEnd);
}

Here is the call graph for this function:

Here is the caller graph for this function:

Call this to copy bytes out of the scanner that have not yet been consumed by the tokenization process.

call this to copy bytes out of the scanner that have not yet been consumed by the tokenization process.

gess 5/12/98

Parameters:
aCopyBufferis where the scanner buffer will be copied to
Returns:
nada

Definition at line 1451 of file nsScanner.cpp.

                                                    {
  if (!mSlidingBuffer) {
    aCopyBuffer.Truncate();
    return;
  }

  nsScannerIterator start, end;
  start = mCurrentPosition;
  end = mEndPosition;

  CopyUnicodeTo(start, end, aCopyBuffer);
}

Here is the call graph for this function:

Here is the caller graph for this function:

Definition at line 1342 of file nsScanner.cpp.

{
  aPosition = mCurrentPosition;
}

Here is the caller graph for this function:

Definition at line 1347 of file nsScanner.cpp.

{
  aPosition = mEndPosition;
}

Here is the caller graph for this function:

Fill internal buffer with new data, returns an error if no new data is read.

Grab data from underlying stream.

gess4/3/98

gess4/3/98

Returns:
error code

Definition at line 420 of file nsScanner.cpp.

                                   {
  nsresult result=NS_OK;

  if(!mInputStream) {
    result=kEOF;
  }
  else {
    PRUint32 numread=0;
    char buf[kBufsize+1];
    buf[kBufsize]=0;

    // XXX use ReadSegments to avoid extra buffer copy? --darin

    result = mInputStream->Read(buf, kBufsize, &numread);
    if (0 == numread) {
      return kEOF;
    }

    if((0<numread) && NS_SUCCEEDED(result)) {
      AppendASCIItoBuffer(buf, numread, nsnull);
    }
    mTotalRead+=numread;
  }

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

Return the position of the first non-whitespace character.

This is only reliable before consumers start reading from this scanner.

Definition at line 360 of file nsScanner.h.

Here is the caller graph for this function:

retrieve next char from internal input stream

retrieve next char from scanners internal input stream

gess 3/25/98

Parameters:
chis the char to accept new value
Returns:
error code reflecting read status

gess 3/25/98

Parameters:
@returnerror code reflecting read status

Definition at line 454 of file nsScanner.cpp.

                                            {
  nsresult result=NS_OK;
  aChar=0;  

  if (!mSlidingBuffer) {
    return kEOF;
  }

  if (mCurrentPosition == mEndPosition) {
    result = FillBuffer();
  }

  if(NS_OK == result){
    aChar = *mCurrentPosition++;
    --mCountRemaining;
  }
  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

Retrieve the name of the file that the scanner is reading from.

In some cases, it's just a given name, because the scanner isn't really reading from a file.

gess 5/12/98

Returns:

Definition at line 1472 of file nsScanner.cpp.

                                     {
  return mFilename;
}

Here is the caller graph for this function:

Internal method used to cause the internal buffer to be filled with data.

gess4/3/98

Definition at line 352 of file nsScanner.h.

{return mIncremental;}

Here is the caller graph for this function:

Records current offset position in input stream.

This allows us to back up to this point if the need should arise, such as when tokenization gets interrupted.

gess 5/12/98

Parameters:
@return

This allows us to back up to this point if the need should arise, such as when tokenization gets interrupted.

gess 7/29/98

Parameters:
@return

Definition at line 295 of file nsScanner.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::Peek ( PRUnichar aChar,
PRUint32  aOffset = 0 
)

peek ahead to consume next char from scanner's internal input buffer

gess 3/25/98

Parameters:
chis the char to accept new value
Returns:
error code reflecting read status

gess 3/25/98

Parameters:
@return

Definition at line 482 of file nsScanner.cpp.

                                                           {
  nsresult result=NS_OK;
  aChar=0;  
  
  if (!mSlidingBuffer) {
    return kEOF;
  }

  if (mCurrentPosition == mEndPosition) {
    result = FillBuffer();
  }

  if(NS_OK == result){
    if (aOffset) {
      while ((NS_OK == result) && (mCountRemaining <= aOffset)) {
        result = FillBuffer();
      }

      if (NS_OK == result) {
        nsScannerIterator pos = mCurrentPosition;
        pos.advance(aOffset);
        aChar=*pos;
      }
    }
    else {
      aChar=*mCurrentPosition;
    }
  }

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::Peek ( nsAString &  aStr,
PRInt32  aNumChars,
PRInt32  aOffset = 0 
)

Definition at line 514 of file nsScanner.cpp.

{
  if (!mSlidingBuffer) {
    return kEOF;
  }

  if (mCurrentPosition == mEndPosition) {
    return FillBuffer();
  }    
  
  nsScannerIterator start, end;

  start = mCurrentPosition;

  if ((PRInt32)mCountRemaining <= aOffset) {
    return kEOF;
  }

  if (aOffset > 0) {
    start.advance(aOffset);
  }

  if (mCountRemaining < PRUint32(aNumChars + aOffset)) {
    end = mEndPosition;
  }
  else {
    end = start;
    end.advance(aNumChars);
  }

  CopyUnicodeTo(start, end, aStr);

  return NS_OK;
}

Here is the call graph for this function:

Consume characters until you run into a char that's not valid in an entity name.

Parameters:
aString- receives new data from stream
Returns:
error code

Definition at line 802 of file nsScanner.cpp.

                                                          {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);
  nsScannerIterator origin, current, end;
  PRBool            found=PR_FALSE;  

  origin = mCurrentPosition;
  current = mCurrentPosition;
  end = mEndPosition;

  while(current != end) {
 
    theChar=*current;
    if(theChar) {
      found=PR_FALSE;
      switch(theChar) {
        case '_':
        case '-':
        case '.':
          // Don't allow ':' in entity names.  See bug 23791
          found = PR_TRUE;
          break;
        default:
          found = ('a'<=theChar && theChar<='z') ||
                  ('A'<=theChar && theChar<='Z') ||
                  ('0'<=theChar && theChar<='9');
          break;
      }

      if(!found) {
        AppendUnicodeTo(mCurrentPosition, current, aString);
        break;
      }
    }
    ++current;
  }
  
  SetPosition(current);
  if (current == end) {
    AppendUnicodeTo(origin, current, aString);
    return FillBuffer();
  }

  //DoErrTest(aString);

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::ReadNumber ( nsString aString,
PRInt32  aBase 
)

Consume digits.

Parameters:
aString- should contain digits
Returns:
error code

Definition at line 861 of file nsScanner.cpp.

                                                              {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  NS_ASSERTION(aBase == 10 || aBase == 16,"base value not supported");

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);
  nsScannerIterator origin, current, end;

  origin = mCurrentPosition;
  current = origin;
  end = mEndPosition;

  PRBool done = PR_FALSE;
  while(current != end) {
    theChar=*current;
    if(theChar) {
      done = (theChar < '0' || theChar > '9') && 
             ((aBase == 16)? (theChar < 'A' || theChar > 'F') &&
                             (theChar < 'a' || theChar > 'f')
                             :PR_TRUE);
      if(done) {
        AppendUnicodeTo(origin, current, aString);
        break;
      }
    }
    ++current;
  }

  SetPosition(current);
  if (current == end) {
    AppendUnicodeTo(origin, current, aString);
    return FillBuffer();
  }

  //DoErrTest(aString);

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

Consume characters until you run into space, a '<', a '>', or a '/'.

Parameters:
aString- receives new data from stream
Returns:
error code

Definition at line 735 of file nsScanner.cpp.

                                                                       {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);
  nsScannerIterator current, end;
  PRBool            found=PR_FALSE;  
  
  current = mCurrentPosition;
  end = mEndPosition;

  // Loop until we find an illegal character. Everything is then appended
  // later.
  while(current != end && !found) {
    theChar=*current;

    switch(theChar) {
      case '\n':
      case '\r':
      case ' ' :
      case '\t':
      case '\v':
      case '\f':
      case '<':
      case '>':
      case '/':
        found = PR_TRUE;
        break;

      case '\0':
        ReplaceCharacter(current, sInvalid);
        break;

      default:
        break;
    }

    if (!found) {
      ++current;
    }
  }

  // Don't bother appending nothing.
  if (current != mCurrentPosition) {
    AppendUnicodeTo(mCurrentPosition, current, aString);
  }

  SetPosition(current);  
  if (current == end) {
    result = FillBuffer();
  }

  //DoErrTest(aString);

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::ReadUntil ( nsAString &  aString,
PRUnichar  aTerminalChar,
PRBool  addTerminal 
)

Consume characters until you find the terminal char.

Consumes chars until you see the given terminalChar.

gess 3/25/98

Parameters:
aStringreceives new data from stream
aTerminalcontains terminating char
addTerminaltells us whether to append terminal to aString
Returns:
error code

gess 3/25/98

Parameters:
@returnerror code

Definition at line 1292 of file nsScanner.cpp.

{
  if (!mSlidingBuffer) {
    return kEOF;
  }

  nsScannerIterator origin, current;

  origin = mCurrentPosition;
  current = origin;

  PRUnichar theChar;
  nsresult result = Peek(theChar);

  if (NS_FAILED(result)) {
    return result;
  }

  while (current != mEndPosition) {
    if (theChar == '\0') {
      ReplaceCharacter(current, sInvalid);
      theChar = sInvalid;
    }

    if (aTerminalChar == theChar) {
      if(addTerminal)
        ++current;
      AppendUnicodeTo(origin, current, aString);
      SetPosition(current);
      return NS_OK;
    }
    ++current;
    theChar = *current;
  }

  // If we are here, we didn't find any terminator in the string and
  // current = mEndPosition
  AppendUnicodeTo(origin, current, aString);
  SetPosition(current);
  return FillBuffer();

}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::ReadUntil ( nsAString &  aString,
const nsReadEndCondition aEndCondition,
PRBool  addTerminal 
)

Consume characters until you find one contained in given terminal set.

Consume characters until you encounter one contained in given input set.

gess 3/25/98

Parameters:
aStringreceives new data from stream
aTermSetcontains set of terminating chars
addTerminaltells us whether to append terminal to aString
Returns:
error code

gess 3/25/98

Parameters:
aStringwill contain the result of this method
aTerminalSetis an ordered string that contains the set of INVALID characters
Returns:
error code

Definition at line 1104 of file nsScanner.cpp.

{  
  if (!mSlidingBuffer) {
    return kEOF;
  }

  nsScannerIterator origin, current;
  const PRUnichar* setstart = aEndCondition.mChars;
  const PRUnichar* setcurrent;

  origin = mCurrentPosition;
  current = origin;

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);

  if (NS_FAILED(result)) {
    return result;
  }
  
  while (current != mEndPosition) {
    theChar = *current;
    if (theChar == '\0') {
      ReplaceCharacter(current, sInvalid);
      theChar = sInvalid;
    }

    // Filter out completely wrong characters
    // Check if all bits are in the required area
    if(!(theChar & aEndCondition.mFilter)) {
      // They were. Do a thorough check.

      setcurrent = setstart;
      while (*setcurrent) {
        if (*setcurrent == theChar) {
          if(addTerminal)
            ++current;
          AppendUnicodeTo(origin, current, aString);
          SetPosition(current);

          //DoErrTest(aString);

          return NS_OK;
        }
        ++setcurrent;
      }
    }
    
    ++current;
  }

  // If we are here, we didn't find any terminator in the string and
  // current = mEndPosition
  SetPosition(current);
  AppendUnicodeTo(origin, current, aString);
  return FillBuffer();
}

Here is the call graph for this function:

nsresult nsScanner::ReadUntil ( nsScannerSharedSubstring aString,
const nsReadEndCondition aEndCondition,
PRBool  addTerminal 
)

Definition at line 1164 of file nsScanner.cpp.

{  
  if (!mSlidingBuffer) {
    return kEOF;
  }

  nsScannerIterator origin, current;
  const PRUnichar* setstart = aEndCondition.mChars;
  const PRUnichar* setcurrent;

  origin = mCurrentPosition;
  current = origin;

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);

  if (NS_FAILED(result)) {
    return result;
  }
  
  while (current != mEndPosition) {
    theChar = *current;
    if (theChar == '\0') {
      ReplaceCharacter(current, sInvalid);
      theChar = sInvalid;
    }

    // Filter out completely wrong characters
    // Check if all bits are in the required area
    if(!(theChar & aEndCondition.mFilter)) {
      // They were. Do a thorough check.

      setcurrent = setstart;
      while (*setcurrent) {
        if (*setcurrent == theChar) {
          if(addTerminal)
            ++current;
          AppendUnicodeTo(origin, current, aString);
          SetPosition(current);

          //DoErrTest(aString);

          return NS_OK;
        }
        ++setcurrent;
      }
    }
    
    ++current;
  }

  // If we are here, we didn't find any terminator in the string and
  // current = mEndPosition
  SetPosition(current);
  AppendUnicodeTo(origin, current, aString);
  return FillBuffer();
}

Here is the call graph for this function:

nsresult nsScanner::ReadUntil ( nsScannerIterator aStart,
nsScannerIterator aEnd,
const nsReadEndCondition aEndCondition,
PRBool  addTerminal 
)

Definition at line 1224 of file nsScanner.cpp.

{
  if (!mSlidingBuffer) {
    return kEOF;
  }

  nsScannerIterator origin, current;
  const PRUnichar* setstart = aEndCondition.mChars;
  const PRUnichar* setcurrent;

  origin = mCurrentPosition;
  current = origin;

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);
  
  if (NS_FAILED(result)) {
    aStart = aEnd = current;
    return result;
  }
  
  while (current != mEndPosition) {
    if (theChar == '\0') {
      ReplaceCharacter(current, sInvalid);
      theChar = sInvalid;
    }

    // Filter out completely wrong characters
    // Check if all bits are in the required area
    if(!(theChar & aEndCondition.mFilter)) {
      // They were. Do a thorough check.
      setcurrent = setstart;
      while (*setcurrent) {
        if (*setcurrent == theChar) {
          if(addTerminal)
            ++current;
          aStart = origin;
          aEnd = current;
          SetPosition(current);

          return NS_OK;
        }
        ++setcurrent;
      }
    }
    
    ++current;
    theChar = *current;
  }

  // If we are here, we didn't find any terminator in the string and
  // current = mEndPosition
  SetPosition(current);
  aStart = origin;
  aEnd = current;
  return FillBuffer();
}

Here is the call graph for this function:

nsresult nsScanner::ReadWhile ( nsString aString,
nsString aValidSet,
PRBool  addTerminal 
)

Consume characters while they're members of anInputSet.

Consume chars as long as they are in the given validSet of input chars.

gess 3/25/98

Parameters:
aStringreceives new data from stream
anInputSetcontains valid chars
addTerminaltells us whether to append terminal to aString
Returns:
error code

gess 3/25/98

Parameters:
aStringwill contain the result of this method
aValidSetis an ordered string that contains the valid characters
Returns:
error code

Definition at line 1047 of file nsScanner.cpp.

                                                {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar         theChar=0;
  nsresult          result=Peek(theChar);
  nsScannerIterator origin, current, end;

  origin = mCurrentPosition;
  current = origin;
  end = mEndPosition;

  while(current != end) {
 
    theChar=*current;
    if (theChar == '\0') {
      ReplaceCharacter(current, sInvalid);
      theChar = sInvalid;
    }
    if(theChar) {
      PRInt32 pos=aValidSet.FindChar(theChar);
      if(kNotFound==pos) {
        if(addTerminal)
          ++current;
        AppendUnicodeTo(origin, current, aString);
        break;
      }
    }
    ++current;
  }

  SetPosition(current);
  if (current == end) {
    AppendUnicodeTo(origin, current, aString);
    return FillBuffer();
  }

  //DoErrTest(aString);

  return result;

}

Here is the call graph for this function:

nsresult nsScanner::ReadWhitespace ( nsScannerSharedSubstring aString,
PRInt32 aNewlinesSkipped,
PRBool aHaveCR 
)

Consume characters until you find the terminal char.

gess 3/25/98

Parameters:
aStringreceives new data from stream
addTerminaltells us whether to append terminal to aString
Returns:
error code

Definition at line 912 of file nsScanner.cpp.

                                                    {

  aHaveCR = PR_FALSE;

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar theChar = 0;
  nsresult  result = Peek(theChar);
  
  if (NS_FAILED(result)) {
    return result;
  }
  
  nsScannerIterator origin, current, end;
  PRBool done = PR_FALSE;  

  origin = mCurrentPosition;
  current = origin;
  end = mEndPosition;

  PRBool haveCR = PR_FALSE;

  while(!done && current != end) {
    switch(theChar) {
      case '\n':
      case '\r':
        {
          ++aNewlinesSkipped;
          PRUnichar thePrevChar = theChar;
          theChar = (++current != end) ? *current : '\0';
          if ((thePrevChar == '\r' && theChar == '\n') ||
              (thePrevChar == '\n' && theChar == '\r')) {
            theChar = (++current != end) ? *current : '\0'; // CRLF == LFCR => LF
            haveCR = PR_TRUE;
          } else if (thePrevChar == '\r') {
            // Lone CR becomes CRLF; callers should know to remove extra CRs
            AppendUnicodeTo(origin, current, aString);
            aString.writable().Append(PRUnichar('\n'));
            origin = current;
            haveCR = PR_TRUE;
          }
        }
        break;
      case ' ' :
      case '\t':
        theChar = (++current != end) ? *current : '\0';
        break;
      default:
        done = PR_TRUE;
        AppendUnicodeTo(origin, current, aString);
        break;
    }
  }

  SetPosition(current);
  if (current == end) {
    AppendUnicodeTo(origin, current, aString);
    result = FillBuffer();
  }

  aHaveCR = haveCR;
  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

nsresult nsScanner::ReadWhitespace ( nsScannerIterator aStart,
nsScannerIterator aEnd,
PRInt32 aNewlinesSkipped 
)

Definition at line 982 of file nsScanner.cpp.

                                                              {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar theChar = 0;
  nsresult  result = Peek(theChar);
  
  if (NS_FAILED(result)) {
    return result;
  }
  
  nsScannerIterator origin, current, end;
  PRBool done = PR_FALSE;  

  origin = mCurrentPosition;
  current = origin;
  end = mEndPosition;

  while(!done && current != end) {
    switch(theChar) {
      case '\n':
      case '\r': ++aNewlinesSkipped;
      case ' ' :
      case '\t':
        {
          PRUnichar thePrevChar = theChar;
          theChar = (++current != end) ? *current : '\0';
          if ((thePrevChar == '\r' && theChar == '\n') ||
              (thePrevChar == '\n' && theChar == '\r')) {
            theChar = (++current != end) ? *current : '\0'; // CRLF == LFCR => LF
          }
        }
        break;
      default:
        done = PR_TRUE;
        aStart = origin;
        aEnd = current;
        break;
    }
  }

  SetPosition(current);
  if (current == end) {
    aStart = origin;
    aEnd = current;
    result = FillBuffer();
  }

  return result;
}

Here is the call graph for this function:

Definition at line 1379 of file nsScanner.cpp.

{
  if (mSlidingBuffer) {
    mSlidingBuffer->ReplaceCharacter(aPosition, aChar);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Resets current offset position of input stream to marked position.

This allows us to back up to this point if the need should arise, such as when tokenization gets interrupted. NOTE: IT IS REALLY BAD FORM TO CALL RELEASE WITHOUT CALLING MARK FIRST!

gess 5/12/98

Parameters:
@return

Definition at line 278 of file nsScanner.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

void nsScanner::SelfTest ( void  ) [static]

Conduct self test.

Actually, selftesting for this class occurs in the parser selftest.

gess 3/25/98

Parameters:
@return

Definition at line 1485 of file nsScanner.cpp.

                             {
#ifdef _DEBUG
#endif
}
nsresult nsScanner::SetDocumentCharset ( const nsACString &  aCharset,
PRInt32  aSource 
)

Use this setter to change the scanner's unicode decoder.

ftang 3/02/99

Parameters:
aCharseta normalized (alias resolved) charset name
aCharsetSource-where the charset info came from
Returns:

Definition at line 198 of file nsScanner.cpp.

                                                                                   {

  nsresult res = NS_OK;

  if( aSource < mCharsetSource) // priority is lower the the current one , just
    return res;

  nsCOMPtr<nsICharsetAlias> calias(do_GetService(kCharsetAliasCID, &res));
  NS_ASSERTION( nsnull != calias, "cannot find charset alias");
  if( NS_SUCCEEDED(res) && (nsnull != calias))
  {
    PRBool same = PR_FALSE;
    res = calias->Equals(aCharset, mCharset, &same);
    if(NS_SUCCEEDED(res) && same)
    {
      return NS_OK; // no difference, don't change it
    }
    // different, need to change it
    nsCAutoString charsetName;
    res = calias->GetPreferred(aCharset, charsetName);

    if(NS_FAILED(res) && (kCharsetUninitialized == mCharsetSource) )
    {
       // failed - unknown alias , fallback to ISO-8859-1
      charsetName.AssignLiteral("ISO-8859-1");
    }
    mCharset = charsetName;
    mCharsetSource = aSource;

    nsCOMPtr<nsICharsetConverterManager> ccm = 
             do_GetService(kCharsetConverterManagerCID, &res);
    if(NS_SUCCEEDED(res) && (nsnull != ccm))
    {
      nsIUnicodeDecoder * decoder = nsnull;
      res = ccm->GetUnicodeDecoderRaw(mCharset.get(), &decoder);
      if(NS_SUCCEEDED(res) && (nsnull != decoder))
      {
         NS_IF_RELEASE(mUnicodeDecoder);

         mUnicodeDecoder = decoder;
      }    
    }
  }
  return res;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void nsScanner::SetIncremental ( PRBool  anIncrValue) [inline]

Definition at line 353 of file nsScanner.h.

{mIncremental=anIncrValue;}

Here is the caller graph for this function:

void nsScanner::SetParser ( nsParser aParser) [inline]

Definition at line 365 of file nsScanner.h.

      {
        mParser = aParser;
      }

Here is the caller graph for this function:

void nsScanner::SetPosition ( nsScannerIterator aPosition,
PRBool  aTruncate = PR_FALSE,
PRBool  aReverse = PR_FALSE 
)

Definition at line 1352 of file nsScanner.cpp.

{
  if (mSlidingBuffer) {
#ifdef DEBUG
    PRUint32 origRemaining = mCountRemaining;
#endif

    if (aReverse) {
      mCountRemaining += (Distance(aPosition, mCurrentPosition));
    }
    else {
      mCountRemaining -= (Distance(mCurrentPosition, aPosition));
    }

    NS_ASSERTION((mCountRemaining >= origRemaining && aReverse) ||
                 (mCountRemaining <= origRemaining && !aReverse),
                 "Improper use of nsScanner::SetPosition. Make sure to set the"
                 " aReverse parameter correctly");

    mCurrentPosition = aPosition;
    if (aTerminate && (mCurrentPosition == mEndPosition)) {
      mMarkPosition = mCurrentPosition;
      mSlidingBuffer->DiscardPrefix(mCurrentPosition);
    }
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Skip over chars as long as they're in aSkipSet.

gess 3/25/98

Parameters:
setof chars to be skipped
Returns:
error code

gess 3/25/98

Parameters:
aSkipSetis an ordered string.
Returns:
error code

Definition at line 643 of file nsScanner.cpp.

                                              {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar theChar=0;
  nsresult  result=NS_OK;

  while(NS_OK==result) {
    result=Peek(theChar);
    if(NS_OK == result) {
      PRInt32 pos=aSkipSet.FindChar(theChar);
      if(kNotFound==pos) {
        break;
      }
      GetChar(theChar);
    } 
    else break;
  } //while
  return result;

}

Here is the call graph for this function:

Here is the caller graph for this function:

Skip over chars as long as they equal given char.

gess 3/25/98

Parameters:
charto be skipped
Returns:
error code

gess 3/25/98

Parameters:
@returnerror code

Definition at line 613 of file nsScanner.cpp.

                                               {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar ch=0;
  nsresult   result=NS_OK;

  while(NS_OK==result) {
    result=Peek(ch);
    if(NS_OK == result) {
      if(ch!=aSkipChar) {
        break;
      }
      GetChar(ch);
    } 
    else break;
  } //while
  return result;

}

Here is the call graph for this function:

Skip over chars as long as they're in aSequence.

Skip over chars as long as they're in aValidSet.

gess 3/25/98

Parameters:
containssequence to be skipped
Returns:
error code

gess 3/25/98

Parameters:
aValidSetis an ordered string containing the characters you want to skip
Returns:
error code

Definition at line 724 of file nsScanner.cpp.

                                               {
  NS_NOTYETIMPLEMENTED("Error: SkipPast not yet implemented.");
  return NS_OK;
}

Skip over chars until they're in aValidSet.

gess 3/25/98

Parameters:
aValidset contains chars you're looking for
Returns:
error code

gess 3/25/98

Parameters:
aValidset is an ordered string that contains chars you're looking for
Returns:
error code

Definition at line 676 of file nsScanner.cpp.

                                             {
  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar ch=0;
  nsresult  result=NS_OK;

  while(NS_OK==result) {
    result=Peek(ch);
    if(NS_OK == result) {
      PRInt32 pos=aValidSet.FindChar(ch);
      if(kNotFound!=pos) {
        break;
      }
      GetChar(ch);
    } 
    else break;
  } //while
  return result;
}

Here is the call graph for this function:

nsresult nsScanner::SkipWhitespace ( PRInt32 aNewlinesSkipped)

Skip whitespace on scanner input stream.

gess 3/25/98

Returns:
error status

gess 3/25/98

Parameters:
@returnerror status

Definition at line 557 of file nsScanner.cpp.

                                                            {

  if (!mSlidingBuffer) {
    return kEOF;
  }

  PRUnichar theChar = 0;
  nsresult  result = Peek(theChar);
  
  if (NS_FAILED(result)) {
    return result;
  }
  
  nsScannerIterator current = mCurrentPosition;
  PRBool    done = PR_FALSE;
  PRBool    skipped = PR_FALSE;
  
  while (!done && current != mEndPosition) {
    switch(theChar) {
      case '\n':
      case '\r': ++aNewlinesSkipped;
      case ' ' :
      case '\t':
        {
          skipped = PR_TRUE;
          PRUnichar thePrevChar = theChar;
          theChar = (++current != mEndPosition) ? *current : '\0';
          if ((thePrevChar == '\r' && theChar == '\n') ||
              (thePrevChar == '\n' && theChar == '\r')) {
            theChar = (++current != mEndPosition) ? *current : '\0'; // CRLF == LFCR => LF
          }
        }
        break;
      default:
        done = PR_TRUE;
        break;
    }
  }

  if (skipped) {
    SetPosition(current);
    if (current == mEndPosition) {
      result = FillBuffer();
    }
  }

  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

PRBool nsScanner::UngetReadable ( const nsAString &  aBuffer)

harishd 01/12/99

Insert data to our underlying input buffer as if it were read from an input stream.

Parameters:
@return

harishd 01/12/99

Returns:
error code

Definition at line 311 of file nsScanner.cpp.

                                                        {
  if (!mSlidingBuffer) {
    return PR_FALSE;
  }

  mSlidingBuffer->UngetReadable(aBuffer,mCurrentPosition);
  mSlidingBuffer->BeginReading(mCurrentPosition); // Insertion invalidated our iterators
  mSlidingBuffer->EndReading(mEndPosition);
 
  PRUint32 length = aBuffer.Length();
  mCountRemaining += length; // Ref. bug 117441
  mTotalRead += length;
  return PR_TRUE;
}

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Definition at line 403 of file nsScanner.h.

Definition at line 402 of file nsScanner.h.

Definition at line 397 of file nsScanner.h.

Definition at line 393 of file nsScanner.h.

Definition at line 395 of file nsScanner.h.

Definition at line 396 of file nsScanner.h.

Definition at line 401 of file nsScanner.h.

Definition at line 400 of file nsScanner.h.

Definition at line 391 of file nsScanner.h.

Definition at line 394 of file nsScanner.h.

Definition at line 405 of file nsScanner.h.

Definition at line 392 of file nsScanner.h.

Definition at line 399 of file nsScanner.h.

Definition at line 404 of file nsScanner.h.


The documentation for this class was generated from the following files: