TOKENINIT()
 Initializes a string for TOKENNEXT()
------------------------------------------------------------------------------
 Syntax

     TOKENINIT([<cString>,[<cDelimiter>],
        [<nSkipDistance>]]) --> lStatus

 Arguments

     <cString>  [@]  Designates a character string for which tokenizing
     is initialized.  This must be passed by reference!

     <cDelimiter>  Designates a list (character string) of individual
     delimiters for tokenizing.

     <nSkipDistance>  Designates the number of delimiting
     characters/sequences after which a value, or a null string (if
     necessary), is returned.  The default value indicates that these
     characters/sequences are not counted.

     ()  When called with no parameters, TOKENNEXT() is set to begin at the
     start of the string.

 Returns

     The function returns .F. when the string variable cannot be initialized.
     For example, the function returns .F. if the string variable was not
     passed by reference.

 Description

     When used in conjunction with the TOKENNEXT() function, an extremely
     versatile incremental tokenizer is available to you.  Specific
     separation processes can be implemented much more quickly than with the
     group of functions around TOKEN().  The speed increase is achieved in
     two ways.

     TOKENINIT() exchanges all delimiting characters for the first one in the
     delimiter list.  This means the entire delimiter list does not have to
     be searched every time.  The second advantage is that TOKENNEXT() does
     not always begin its search for the token that is extracted at the
     beginning of the string (see the function description for TOKENNEXT()).
     However, in contrast to TOKEN(), TOKENNEXT() is unable to extract a
     specific token.

     You can also use the third parameter, a skip distance for the delimiter
     characters.  This allows recognition of empty lines within a text.  In
     this case, an empty line would be displayed by a CrLfCrLf sequence.
     Since TOKENINIT() takes all designated delimiting characters and
     exchanges them for something uniform, this sequence is changed to
     CrCrCrCr.  A skip distance of 2 means that the two delimiters (in this
     case, Cr)) return a token each time.  Since nothing precedes this
     example, TOKENNEXT() returns a null string.

     The function uses the following list of delimiters as standard:

     CHR 32, 0, 9, 10, 12, 26, 32, 138, 141

     and the characters ,.;:!?/\<<>>()^#&%+-*

     This list can be replaced by your own delimiter list, <cDelimiter>.
     Here are some examples of meaningful delimiting characters:

     Table 4-5: Recommended Delimiting Sequences
     ------------------------------------------------------------------------
     Description         <cDelimiter>
     ------------------------------------------------------------------------
     Pages               CHR(12)(Form Feed)
     Sentence            ".!?"
     File Name           ":\."
     Numeric strings     ",."
     Date strings        "/."
     Time strings        ":."
     ------------------------------------------------------------------------

 Notes

     .  When using a skip value, you must use the TOKENEND() function
        as a loop condition.

     .  When TOKENINIT() exchanges all delimiting characters for a new
        one, the first delimiter on the list is always used.  This ensures
        that no character contained in the token becomes a delimiter.

     .  When you use the TOKENINIT() or TOKENNEXT(), you cannot use
        the TOKENSEP() function.  The required information can be determined
        using TOKENAT() in conjunction with the original string (status
        before TOKENINIT()).

           To determine the delimiter position before the last token, set
           TOKENAT() to -1.  To determine the delimiter position after the
           last token, set TOKENAT to .T..

 Examples

     .  Break a string into words.  First the string must be
        initialized:

        cDelim   :=  "!?.,-"
        cString   :=  "A.B-C,D!E??"

        TOKENINIT(@cString, cDelim)         // "A!B!C!D!E!!"


        Do While .NOT. TOKENEND()
           cWord  :=  TOKENNEXT(cString)
           ? cWord
        ENDDO

     .  Break text into lines.  Take blank lines into account using a
        skip distance of 2:

        nCounter  :=  0

        TOKENINIT(cTextString, CHR(13) + CHR(10), 2)

        DO WHILE .NOT. TOKENEND()
           nLine  :=  TOKENNEXT(cTextString)
           ++ nCounter
        ENDDO

        ? nCounter