#1
  1. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2003
    Location
    Chicago
    Posts
    14
    Rep Power
    0

    How to continiously read from stdin and parse string.


    Hi,

    I am confused as how to go about doing this.
    This is the problem....

    I have a standard in as in Unix

    cat accesslog | ./sample.out

    in sample.c file I need to read the records into a string in a while loop and parse the string.

    The string format is

    fixedfield1 \s fixedfield2\s fixedfield3\s fixedfield4\s fixedfield5\s variablefield1\s variablefield2\s variablefield3\s ....

    The variable field I need to put it into one variable in the program.


    I am coding this in C and if you can give a sample template as to how to read in a while loop and parse .......that would be great.


    Thanks in advance.
  2. #2
  3. I'm Baaaaaaack!
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Jul 2003
    Location
    Maryland
    Posts
    5,538
    Rep Power
    243
    I am not sure what you are looking for. You can read from stdin exactly like you would read from a normal file descriptor you opened yourself: on character at a time, a buffer's worth at a time, or with fscanf.

    Perhaps with some example input I can be more help.

    My blog, The Fount of Useless Information http://sol-biotech.com/wordpress/
    Free code: http://sol-biotech.com/code/.
    Secure Programming: http://sol-biotech.com/code/SecProgFAQ.html.
    Performance Programming: http://sol-biotech.com/code/PerformanceProgramming.html.
    LinkedIn Profile: http://www.linkedin.com/in/keithoxenrider

    It is not that old programmers are any smarter or code better, it is just that they have made the same stupid mistake so many times that it is second nature to fix it.
    --Me, I just made it up

    The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.
    --George Bernard Shaw
  4. #3
  5. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2003
    Location
    Chicago
    Posts
    14
    Rep Power
    0

    reading from stdin elaborated


    I have this problem. I have this log file which is continiously written. I tail out into another process to parse these records and write a tab delimited file.

    here is the command

    tail -f /logs/access_log | ./parser.o

    In the parser.c I have this code

    ....
    ....
    ....
    while(endFlg == 0)
    {
    ...
    ...
    ...
    do
    {
    // If End of file is reached break from the do...while loop
    ptmp = fgets(atmp, sizeof(atmp), stdin);
    if(ptmp == NULL)
    {
    endFlg = 1;
    break;
    }
    //Initialize the structure
    printf("%s\n","Start 2nd.............");
    memset(fileStr,'\0',sizeof(fileStr));
    memset(&logentry,0,sizeof(logentry));
    ...
    ...
    }while ( time < 300)
    ....
    ....
    }


    Now the problem is not all the records from the log file gets logged. It almost skips 1/3 of the records.If I have 150 records in /logs/access_log I get only 50 output records

    Where does it store the record before reading it into the process...Am I missing some of the records as I am parsing....???

    This is in C coding
    Thanks in Advance
  6. #4
  7. I'm Baaaaaaack!
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Jul 2003
    Location
    Maryland
    Posts
    5,538
    Rep Power
    243
    First, check the output from tail and be sure it is what you expect. Second, run your program without parsing and dump stdin to file to see if the log file and what your process sees are identical. Third, if you are capable of modifying the process that is writting the log file, why not do the parsing there? Fourth, include your own version of tail in your parser program. In order to do that, look up ftell. You can use that to check the end location of the file and each time it increases process the extra. Be sure to put in something to quite after so much time has elapsed with no increase in length!

    BTW: please enclose your code in brackets (see http://forums.devshed.com/misc.php?action=bbcode&s=).
    Last edited by mitakeet; August 5th, 2003 at 04:57 PM.

    My blog, The Fount of Useless Information http://sol-biotech.com/wordpress/
    Free code: http://sol-biotech.com/code/.
    Secure Programming: http://sol-biotech.com/code/SecProgFAQ.html.
    Performance Programming: http://sol-biotech.com/code/PerformanceProgramming.html.
    LinkedIn Profile: http://www.linkedin.com/in/keithoxenrider

    It is not that old programmers are any smarter or code better, it is just that they have made the same stupid mistake so many times that it is second nature to fix it.
    --Me, I just made it up

    The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.
    --George Bernard Shaw
  8. #5
  9. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2003
    Location
    Chicago
    Posts
    14
    Rep Power
    0

    thanks


    I got it. I researched ftell. Before I make the change to the code just want to run by and see if what you said is ok.

    I read the log file till the end....do a ftell on it and close it.

    Sleep for sometime and reopen the log file.

    fseek to the previous position and read the log file again till the end.

    If a sample code would be great ,,


    Thanks for the suggestions...
  10. #6
  11. I'm Baaaaaaack!
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Jul 2003
    Location
    Maryland
    Posts
    5,538
    Rep Power
    243

    Example (works on Win2K)


    I hard coded a lot of stuff, but this worked fine when I tested it...

    Code:
    #include <stdio.h>
    #include <stdlib.h>
    
    #include <windows.h> /* required for Sleep() */
    
    #define MAXLINELENGTH 4096
    
    /* this is a dummy that just prints, put your own code here */
    void processLine(char * line){
        printf("%s", line);
        return;
    }
    
    int main(){
        char fileName[] = "test.txt";
        char input[MAXLINELENGTH];
        FILE *fin;
        long telPos = 0;
        int len;
    
        if ((fin = fopen(fileName, "r")) == NULL){
            fprintf(stderr, "Cannot open %s!\n", fileName);
            exit(1);
        }
    
        for (;;){
            if (fseek(fin, telPos, SEEK_SET)){
                fprintf(stderr, "Error in fseek at telPos %d!\n", telPos);
                exit(1);
            }
            while (fgets(input, MAXLINELENGTH, fin)){
                len = strlen(input);
                if (input[len-1] != '\n') break; /* we did not find a full record */
                processLine(input);
                telPos = ftell(fin);
            }
            if (feof(fin)){
                clearerr(fin);
                Sleep(100);
            }
        }
    
        fclose(fin);/* never going to happen in this code */
    
        return 0;
    
    }

    My blog, The Fount of Useless Information http://sol-biotech.com/wordpress/
    Free code: http://sol-biotech.com/code/.
    Secure Programming: http://sol-biotech.com/code/SecProgFAQ.html.
    Performance Programming: http://sol-biotech.com/code/PerformanceProgramming.html.
    LinkedIn Profile: http://www.linkedin.com/in/keithoxenrider

    It is not that old programmers are any smarter or code better, it is just that they have made the same stupid mistake so many times that it is second nature to fix it.
    --Me, I just made it up

    The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.
    --George Bernard Shaw
  12. #7
  13. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2003
    Location
    Chicago
    Posts
    14
    Rep Power
    0

    Thanks


    That was a very extensive and detailed code.

    Thanks for the code... got it to work.

IMN logo majestic logo threadwatch logo seochat tools logo