#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0

    Exact behaviour of strtok


    #include <stdio.h>
    #include <string.h>

    int main( int argc, char **argv)
    {
    char str[] ="helloh:howh:are:you;";
    char *tok;
    int no = 0;

    tok = strtok(str,":h");

    while(tok) {
    printf("token %d : %s\n", ++no,tok);
    tok = strtok(NULL,":y");
    }

    printf("str : %s\n",str);
    return 0;
    }


    Output :
    token 1 : ello
    token 2 : howh
    token 3 : are
    token 4 : ou;
    str : hello

    This is from Man Page:

    " A sequence of two or more contiguous delimiter characters in the parsed string is considered to be a single delimiter. Delimiter characters at the start or end of the string are ignored. "

    when the sequence of two or more characters are treated as a single delimiter the output should contain strings other than the delimiting characters.

    The output i am expecting from the strtok is as below.

    helloh
    owh:are:you;

    But the actual output is different.

    Whats the story..?
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Aug 2012
    Posts
    156
    Rep Power
    33
    Quote from http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtok.html
    The first call in the sequence searches the string pointed to by s1 for the first byte that is not contained in the current separator string pointed to by s2. If no such byte is found, then there are no tokens in the string pointed to by s1 and strtok() shall return a null pointer. If such a byte is found, it is the start of the first token.

    The strtok() function then searches from there for a byte that is contained in the current separator string. If no such byte is found, the current token extends to the end of the string pointed to by s1, and subsequent searches for a token shall return a null pointer. If such a byte is found, it is overwritten by a NUL character, which terminates the current token. The strtok() function saves a pointer to the following byte, from which the next search for a token shall start.
    First strtok: replace the second 'h' with NUL
    Code:
    /* 'h', 'e', 'l', 'l', 'o', NUL, ':', 'h', 'o', 'w', 'h', ':', 'a', 'r', 'e', ':', 'y', 'o', 'u', ';' */
    /*      ^ tok                         ^ next */
    Second strtok (inside the loop): starting at the now first 'h' (previously third 'h'), search for characters 'y' or ':', replacing the now first ':' with NUL
    Code:
    /* 'h', 'e', 'l', 'l', 'o', NUL, ':', 'h', 'o', 'w', 'h', NUL, 'a', 'r', 'e', ':', 'y', 'o', 'u', ';' */
    /*                                    ^ tok                     ^ next */
    Summary: for each strtok call only 1 NUL (at most) is written.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0
    Originally Posted by bdb
    Quote from http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtok.html


    First strtok: replace the second 'h' with NUL
    Code:
    /* 'h', 'e', 'l', 'l', 'o', NUL, ':', 'h', 'o', 'w', 'h', ':', 'a', 'r', 'e', ':', 'y', 'o', 'u', ';' */
    /*      ^ tok                         ^ next */
    Second strtok (inside the loop): starting at the now first 'h' (previously third 'h'), search for characters 'y' or ':', replacing the now first ':' with NUL
    Code:
    /* 'h', 'e', 'l', 'l', 'o', NUL, ':', 'h', 'o', 'w', 'h', NUL, 'a', 'r', 'e', ':', 'y', 'o', 'u', ';' */
    /*                                    ^ tok                     ^ next */
    Summary: for each strtok call only 1 NUL (at most) is written.

    Thanks for reply.

    1)Now can i say that strtok function prints the tokens( or words) from another string by removing the delimiters.

    below is the code i have written that exactly works as strtok
    except it does not follow the same signature as strtok.

    #include <stdio.h>
    #include <string.h>


    void Str_Tok(char *Mstr, char *pDlm);

    int main( int argc, char **argv)
    {
    char str[] = "helloh:howh:are:you;";
    char *dlim = "o;";
    char *tok;

    printf("str : %s and dlim : %s\n",str,dlim);

    printf("\n user\n");
    Str_Tok(str,dlim);

    printf(" \n lib\n");
    tok = strtok(str,dlim);
    while(tok){
    printf(" %s \n", tok);
    tok =strtok(NULL,dlim);
    }

    return 0;
    }


    void Str_Tok(char *Mstr, char *pDlm)
    {
    char *Dlm = pDlm;
    int tok_fnd = 0;
    char t[20];
    char *token = t;

    while(*Mstr ){

    while(*Dlm && (*Dlm != *Mstr))
    Dlm++;

    if(*Dlm){
    *Mstr++;
    if ( tok_fnd ){
    *token = '\0';

    printf(" %s\n", t);
    tok_fnd = 0,token = t;
    }
    }else {
    tok_fnd = 1;
    *token++ =*Mstr++;
    }

    Dlm = pDlm;
    }
    *token = '\0';
    printf(" %s\n", t);
    return ;
    }

    inorder to write strtok kind of function i think we need to use dynamic memory allocation to retain the main string between calls.

    any idea how strtok is implemented in library function.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Aug 2012
    Posts
    156
    Rep Power
    33
    Originally Posted by Chead
    inorder to write strtok kind of function i think we need to use dynamic memory allocation to retain the main string between calls.
    Not really. A static pointer to the start of the next token is enough (mind reentrancy!!!!).

    [code=c]char *my_strtok(char *text, const char *delims) {
    static char *next;
    char *last, *prev;
    if (text) next = text; /* first call */
    next += strspn(next, delims);
    prev = next;
    last = next + strcspn(next, delims);
    if (last == next) return NULL; /* no more tokens */
    *last = 0;
    next = last + 1;
    return prev;
    }[/code]

    Comments on this post

    • Chead agrees : Good
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0
    Originally Posted by bdb
    Not really. A static pointer to the start of the next token is enough (mind reentrancy!!!!).

    [code=c]char *my_strtok(char *text, const char *delims) {
    static char *next;
    char *last, *prev;
    if (text) next = text; /* first call */
    next += strspn(next, delims);
    prev = next;
    last = next + strcspn(next, delims);
    if (last == next) return NULL; /* no more tokens */
    *last = 0;
    next = last + 1;
    return prev;
    }[/code]
    Thanks very much for code, i actually came to know two new function .. :)
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    13
    Rep Power
    0
    Code:
    char *st(char *Mstr , const char *pDlm)
    {
        static char *next;
        char *start=NULL , *end=NULL;
        const char *Dlm = pDlm;
    
        int fnd = 0;
        if (Mstr)
            next = Mstr;
    
        while ( next && *next ){
            while(*Dlm && (*Dlm != *next))
                Dlm++;
            if(*Dlm){
                if(fnd){
                    end = next;
                    *end = '\0';
                    next++;
                    return start;
                }
                }else if (fnd == 0){
                        fnd = 1 ;
                    start = next;
    
                }
            next++;Dlm = pDlm;
            }
        if(next && !*next) {
            end = next;
            *end = '\0';
            if(start)
                return start;
        }
        return NULL;
    }

    I have written a code which behaves same as standard functions.
    is the standard function implemented in the same way.
    please let me know if any optimized version is available.

IMN logo majestic logo threadwatch logo seochat tools logo