#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    38
    Rep Power
    6

    How to execute a script that takes hours


    I need to import some products into a MySQL DB
    new products code are push from an external script to a web service on my server
    once I collect new products code that need to be imported each day I need to check them against a SOAP web service to retrive some additional data

    the soap web service is pretty slow and can take up to 3/5 second each time and I may have 2000 precuts to import each day
    It can basically take up to 3 hours to scan all the products

    After I get the extra data from i can create SQL statement needed to add the product into MySQL database
    I guess that the bast thing is to store them in a .sql file or DB and execute them at the end

    so

    STEP 1
    - web service push product code that need to be imported to my script
    - my script writes the product code to database

    STEP 2
    *** a script loads new codes that needs to be imported and for each one make a SOAP call to get additional data
    - I can create iNSERT statement for my my MySQL DB

    STEP 3
    - execute MySQL statements

    the problem is how can i execute and loop all products i need to import without experiencing timeout
  2. #2
  3. Impoverished Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,862
    Rep Power
    9646
    Make your web service store the information it receives in a sort of "pending" table in your database. That's all it does.

    Then make a command-line script which reads from that table and does its work. Have the script end itself after, say, 9 minutes. Make the script start with a cronjob that runs every 10 minutes. The exact times aren't that important but the script should end itself before the next one starts.

    That way you have this script running automatically every once in a while to process whatever was in that pending table.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    38
    Rep Power
    6
    the STEP 1 script is the script that import data into the "pending" table

    I should add the STEP 2 script to a cron job and be executed every 15 minutes, and set inside this script a set_time_limit(600); // 10 minutes
    is it correct?
  6. #4
  7. Impoverished Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,862
    Rep Power
    9646
    Don't use set_time_limit() as that will kill your script after the time has passed. Even if it's in the middle of doing something. You need to end it cleanly, and that means watching the time yourself.
    PHP Code:
    $end time() + 600// 10 minutes
    do {

        
    // work...

    } while (time() < $end); 
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    38
    Rep Power
    6
    ok, Thanks!

IMN logo majestic logo threadwatch logo seochat tools logo