#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2000
    Posts
    456
    Rep Power
    15
    Can someone offer some help with this script? It uses DBI to pull URLs from the "url" column of a mysql table. LWP then checks the validity of the URLs. The outcome of the link check is then reflected by an update to the "valid" column. This script worked great for me when I had a few hundred rows in my table. There are now over 7000. After about 900 rows the script just stops executing without any errors.

    What causes the script to suddenly stop before checking every row?

    Thanks!
    Ryan McKillen

    <BLOCKQUOTE><font size="1" face="Verdana,Arial,Helvetica">code:</font><HR><pre>
    #!/usr/bin/perl

    use LWP::UserAgent;
    use DBI;

    $db_database = "filedb";
    $db_uid = "root";
    $db_pwd = "pass";
    ($ua = LWP::UserAgent->new)->timeout(20); #actually set timeout


    $dbh = DBI->connect ("DBI:mysql:$db_database".$mysqlsock, $db_uid, $db_pwd) or die("could not connect to dbn");
    $sth = $dbh->prepare("SELECT url FROM files");
    $sth -> execute();
    $numrows = $sth->rows;

    print "nn";
    while (my $url = $sth->fetchrow_array) {
    if(($ua->request(HTTP::Request->new('HEAD', $url)))->is_success()) {
    $validity = "link works";
    $valid_update = $dbh->do("UPDATE files SET valid = 1 WHERE url = '$url'");
    }
    else {
    $validity = "link sucks";
    $valid_update = $dbh->do("UPDATE files SET valid = valid + 1 WHERE url = '$url'");
    }
    print "$validityn$urlnn";
    }
    $sth->finish;
    [/code]

    [This message has been edited by scream (edited October 10, 2000).]
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Aug 2000
    Location
    Cape Town, WP, South Africa
    Posts
    39
    Rep Power
    15
    It might be you syntax in this line

    $dbh = DBI->connect ("DBI:mysql:$db_database".$mysqlsock, $db_uid, $db_pwd) or die("could not connect to dbn");

    try with qoutes around your password and username.

    ("DBI:mysql:$db_database"."$mysqlsock", "$db_uid, $db_pwd") or use single quotes if this does not work
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Location
    Alkmaar, The Netherlands
    Posts
    123
    Rep Power
    15
    <BLOCKQUOTE><font size="1" face="Verdana,Arial,Helvetica">quote:</font><HR>try with qoutes around your password and username.[/quote]

    That won't work, because Perl won't understand the variables anymore then.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2000
    Posts
    456
    Rep Power
    15
    If it was a little syntax issue involving quotes, perl would generate an error and not work in the first place. That couldn't have anything to with the script just stopping in the middle of its execution.

    This is discouraging! Can anyone else offer me some help?
  8. #5
  9. No Profile Picture
    freebsd
    Guest
    Devshed Newbie (0 - 499 posts)
    Run it from command line and send the output to whatever file.

    You may as well changing the timeout value from 20 to something higher.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2000
    Posts
    456
    Rep Power
    15
    I do run it from the command line and the output just prints to the screen. The timeout value has nothing to do with the problem. If a URL times out it is just counted as invalid. I can't understand what writing the output to a file would do when it already prints to the screen.

    Any more ideas?

IMN logo majestic logo threadwatch logo seochat tools logo