Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7

    Extracting lines


    Hello all
    I have this code
    use File::Tail;
    $|++;
    $name="C:\\users\\Mizo\\desktop\\log.t...
    $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1);
    while (defined($line=$file->read)) {
    if ($line=~/rules.txt/i){
    print "$line";
    }
    }

    Log.txt has this

    Host:xXMizoXx
    Request:blabla
    Method:GET
    Http:HTTP/1.1

    I want the previous code match the rules.txt with only the "Request" line in the log file .. and ignore the others
    do you know how to make it happens?

    And is it possible to take the request line each time and put it in a text file to process it ? and how?
  2. #2
  3. 'fie' on me, allege-dly
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2003
    Location
    in da kitchen ...
    Posts
    12,889
    Rep Power
    6444
    Code:
    open FH, "<rules.txt";
    @rules=<FH>;  # read the rules in to an array
    close FH;
    use File::Tail;
    $|++;
    $name="C:/users/Mizo/desktop/log.txt
    $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1);
    while (defined($line=$file->read)) {
        for (@rules) {
           $rule=$_;
           if ($line=~ m/($rule)/i){
              print "$line";
           }
        }
     }
    Is one way, but if you have a lot of rules, it will be quite inefficient, might be an idea to think about how you could match the atom by splitting the string on ":"
    Code:
    $item=split(":", $line)[0];
    if (index($rules, $item) != -1) {
       print $line;
    }
    --Ax
    without exception, there is no rule ...
    Handmade Irish Jewellery
    Targeted Advertising Cookie Optout (TACO) extension for Firefox
    The great thing about Object Oriented code is that it can make small, simple problems look like large, complex ones


    09 F9 11 02
    9D 74 E3 5B
    D8 41 56 C5
    63 56 88 C0
    Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
    -- Jamie Zawinski
    Detavil - the devil is in the detail, allegedly, and I use the term advisedly, allegedly ... oh, no, wait I did ...
    BIT COINS ANYONE
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2009
    Posts
    3
    Rep Power
    0
    If you need only lines which starts with "Request" then do this:

    print "$line" if $line =~ m/^Request: /;

    But, it is better not to use RE for such easy task. Use index.

    If you need lines that match at least one of the rules from rules.txt then you should change Axweildr's code in such way:

    Code:
    open FH, "<rules.txt";
    @rules=<FH>;  # read the rules in to an array
    close FH;
    use File::Tail;
    $|++;
    $name="C:/users/Mizo/desktop/log.txt
    $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1);
    while (defined($line=$file->read)) {
        for (@rules) {
           $rule=$_;
           if ($line=~ m/($rule)/i){
              print "$line";
              last;
           }
        }
     }

    This last statement prevent from printing one line several times (If it will match several rules).
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Thank you ... it's clear now..
    but one more thing >,<
    The problem is that rules file is in linux
    and the log file is in windows..
    so when i am going to send the agent to windows it's not going to do any matching
    because rules.txt is still in linux
    how to bring the rules file with the agent? from the agent itself.
  8. #5
  9. 'fie' on me, allege-dly
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2003
    Location
    in da kitchen ...
    Posts
    12,889
    Rep Power
    6444
    it should still split on '\r\n' as well as '\n' so it shouldn't be an issue, if you believe it is then you can run unix2dos over the file in question
    --Ax
    without exception, there is no rule ...
    Handmade Irish Jewellery
    Targeted Advertising Cookie Optout (TACO) extension for Firefox
    The great thing about Object Oriented code is that it can make small, simple problems look like large, complex ones


    09 F9 11 02
    9D 74 E3 5B
    D8 41 56 C5
    63 56 88 C0
    Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
    -- Jamie Zawinski
    Detavil - the devil is in the detail, allegedly, and I use the term advisedly, allegedly ... oh, no, wait I did ...
    BIT COINS ANYONE
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Originally Posted by Axweildr
    it should still split on '\r\n' as well as '\n' so it shouldn't be an issue, if you believe it is then you can run unix2dos over the file in question
    No no it's not about spliting
    what i wanted to say is how to send the rules file with the code ...to windows machine beacuase the rules file is in linux..
    and the logfile is in windows
    I want it to go with the agent from linux to windows everytime i run the code
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2009
    Posts
    3
    Rep Power
    0
    Originally Posted by -=Mizo=-
    No no it's not about spliting
    what i wanted to say is how to send the rules file with the code ...to windows machine beacuase the rules file is in Linux..
    and the logfile is in windows
    I want it to go with the agent from Linux to windows everytime i run the code
    I think that it isn't a good idea to send code and rules file to windows from Linux. You should use some mechanism to access log files. For example you can use SSH or FTP or something else to access this files from linux to windows. Another way is to send them periodically from win to linux (by ftp or ssh or sth else).

    You can also share windows folder with log files and mount them on Linux. So it's up to you which way to choose
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,930
    Rep Power
    1225
    Originally Posted by Axweildr
    Code:
    open FH, "<rules.txt";
    @rules=<FH>;  # read the rules in to an array
    close FH;
    use File::Tail;
    $|++;
    $name="C:/users/Mizo/desktop/log.txt
    $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1);
    while (defined($line=$file->read)) {
        for (@rules) {
           $rule=$_;
           if ($line=~ m/($rule)/i){
              print "$line";
           }
        }
     }
    Is one way, but if you have a lot of rules, it will be quite inefficient, might be an idea to think about how you could match the atom by splitting the string on ":"
    Code:
    $item=split(":", $line)[0];
    if (index($rules, $item) != -1) {
       print $line;
    }
    I realize that most of that code is from the OP, but lets take a look at at with Perl Best Practices in mind.

    The script is missing 1, if not 2, very important pragmas which should be in every Perl script.
    Code:
    use warnings;
    use strict;
    It's missing proper error handling on the open call.
    It's better to use the 3 arg form of open and a lexical var for the filehandle.
    Code:
    my $rules_file = 'rules.text';
    open my $FH, '<', $rules_file or die "failed to open '$rules_file' $!";
    The use statements are executed at compile time, so place them at the beginning of the script instead of intermixed with runtime code.

    The script overall is lacking proper horizontal whitespace.

    Please read: perldoc -q quoting

    Code:
        for (@rules) {
           $rule=$_;
           if ($line=~ m/($rule)/i){
              print "$line";
           }
    Is better written as:
    Code:
        for my $rule ( @rules ) {
            print $line and last if $line =~ /$rule/i;
        }
    The gain in efficiency by using index instead of a regex is offset by the split and an if block with only 1 line in the block is better written as 1 line, as shown above. Also, the first arg to split is a pattern (regex) not a string. However, " " is an exception.

    perldoc -f split

    Here's the complete script with a couple adjustments that I didn't mention.
    Code:
    use strict;
    use warnings;
    use File::Tail;
    
    $|++;
    
    my $rules = 'rules.txt';
    
    open my $FH, '<', $rules or die "failed to open '$rules' $!";
    my %rules = map { chomp; lc($_), 1 } <$FH>;
    close $FH;
    
    my $file = File::Tail->new( name        => 'C:/users/Mizo/desktop/log.txt',
                                maxinterval => 1,
                                interval    => 1,
                                adjustafter => 1
    );
    
    while (defined( my $line = $file->read) ) {
    
        my $rule = (split /:/, $line)[0];
        print $line if exists $rules{lc($rule)};
    
    }
    Finally, on the question about accessing the file remotely:
    File::Remote - Read/write/edit remote files transparently
    http://search.cpan.org/~nwiger/File-Remote-1.17/Remote.pm
  16. #9
  17. 'fie' on me, allege-dly
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2003
    Location
    in da kitchen ...
    Posts
    12,889
    Rep Power
    6444
    Thank you for your input
    --Ax
    without exception, there is no rule ...
    Handmade Irish Jewellery
    Targeted Advertising Cookie Optout (TACO) extension for Firefox
    The great thing about Object Oriented code is that it can make small, simple problems look like large, complex ones


    09 F9 11 02
    9D 74 E3 5B
    D8 41 56 C5
    63 56 88 C0
    Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
    -- Jamie Zawinski
    Detavil - the devil is in the detail, allegedly, and I use the term advisedly, allegedly ... oh, no, wait I did ...
    BIT COINS ANYONE
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Thank you all..your answers were very useful.
    but.. it isn't working for me
    Code:
    use strict;
    use warnings;
    use File::Tail;
    $|++;
    my $dir="C:\\users\\Mizo\\desktop\\rules.txt";
    open (FH,$dir) or die $!;
    my @rules=<FH>;  # read the rules in to an array
    close FH or die $!;
    my $name="C:\\users\\Mizo\\desktop\\log.txt";
    my $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1);
    while (defined(my $line=$file->read)) {
        foreach my $rule(@rules) {
           if ($line=~/$rule/i){
              print "$line";
              last;
           }
        }
     }
    it's neither matching nor printing..

    but this code for example is printing when matched..

    Code:
    use File::Tail; 
    $|++;
    $name="C:\\users\\Mizo\\desktop\\log.txt";
    $file=File::Tail->new(name=>$name, maxinterval=>1,interval=>1, adjustafter=>1); 
    while (defined($line=$file->read)) {
    if ($line=~/blabla/i){
    print "$line";
    }
    }
  20. #11
  21. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Code:
    use strict;
    use warnings;
    use File::Tail;
    
    $|++;
    
    my $rules = 'rules.txt';
    
    open my $FH, '<', $rules or die "failed to open '$rules' $!";
    my %rules = map { chomp; lc($_), 1 } <$FH>;
    close $FH;
    
    my $file = File::Tail->new( name        => 'C:/users/Mizo/desktop/log.txt',
                                maxinterval => 1,
                                interval    => 1,
                                adjustafter => 1
    );
    
    while (defined( my $line = $file->read) ) {
    
        my $rule = (split /:/, $line)[0];
        print $line if exists $rules{lc($rule)};
    
    }
    is printing nothing also..
  22. #12
  23. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,930
    Rep Power
    1225
    Please post a few sample lines from rules.txt.

    Is log.txt continuously being updated? If not, then File::Tail is the wrong tool to use in this script.

    The module states:
    File::Tail - Perl extension for reading from continously updated files
  24. #13
  25. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Originally Posted by FishMonger
    Please post a few sample lines from rules.txt.

    Is log.txt continuously being updated? If not, then File::Tail is the wrong tool to use in this script.

    The module states:
    for example..

    [code]
    \.txt\?$
    [\code]
    or even if i put a word in the rules.txt..and request it and the log file saves it .. the script isn't printing anything
    so when i am going to requet with a .txt? in the request it's going to be saved in the log file and when the rules file matchs i will be alerted..

    and yes log.txt is contrinuously updated.
  26. #14
  27. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,930
    Rep Power
    1225
    Based on your sample lines from each file, I don't see why you should expect it to print anything.

    Please post a few sample lines in rules.txt and the corresponding lines in log.txt that should be extracted.
  28. #15
  29. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2008
    Posts
    42
    Rep Power
    7
    Originally Posted by FishMonger
    Based on your sample lines from each file, I don't see why you should expect it to print anything.

    Please post a few sample lines in rules.txt and the corresponding lines in log.txt that should be extracted.
    Code:
    Host:xXMizoXx
    Request:/.txt?
    Method:GET
    Http:HTTP/1.1
    this is the log file...when i request from http webserver
    and i have only 1 rule in rules.txt which i showed u..

    but..
    when i use the old code..
    which is

    if($line=~/http/i){
    print "$line";


    it works..
    once the log file is updated and http is there i get a message
    but the new code isn't working..
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo