1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Rep Power

    Execute process in parallel

    Hello and thank you in advance for any guidance on this question.

    I am fairly new to Python and in the progress of converting some scripts from Powershell. My requirement is to read a CSV file with urls and then execute an external program to verify they are active. The script works great but executes sequentially. I am seeking help to use Python to execute an external program in parallel (for each entry in the csv file) and the capture the return codes which can be used to analysis the url status.

    Here is my current code which works but only sequentially.

    import csv
    import subprocess

    cmd = ("c:\\scripts\\cmd\\http-ping.exe")

    csv_file = csv.DictReader(open(test_file, 'rb'), delimiter=',', quotechar='"')
    for line in csv_file:
    host = line['url']
    print cmd, host
    pname = subprocess.Popen([cmd,host],stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    stdout, stderr = pname.communicate()

    if pname.poll() == 100:
    print "Ok"
    print "Error"

    Here are the results. It executes the first url and then then the second in order.

    C:\scripts>python forum.py
    c:\scripts\cmd\http-ping.exe "http://localhost/xampp/"
    c:\scripts\cmd\http-ping.exe "http://localhost2/xampp/"
  2. #2
  3. Contributing User
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Aug 2011
    Rep Power
    A threaded application is probably good enough. Another choice is multiprocessing. It's not completely simple and straight forward because the display phase (print statements) need to run single threaded to avoid jumbling the output. Offhand, you could use process synchronization; have each thread write to a separate file then concatenate the files after all threads have finished; use a separate thread to print results from a queue which the pinging threads test. I don't have time now to write code.
    [code]Code tags[/code] are essential for python code and Makefiles!

IMN logo majestic logo threadwatch logo seochat tools logo