September 5th, 2013, 12:08 AM
Execute process in parallel
Hello and thank you in advance for any guidance on this question.
I am fairly new to Python and in the progress of converting some scripts from Powershell. My requirement is to read a CSV file with urls and then execute an external program to verify they are active. The script works great but executes sequentially. I am seeking help to use Python to execute an external program in parallel (for each entry in the csv file) and the capture the return codes which can be used to analysis the url status.
Here is my current code which works but only sequentially.
cmd = ("c:\\scripts\\cmd\\http-ping.exe")
test_file = 'C:\SERVER_INVENTORY\SYSTEM_INVENTORY.csv'
csv_file = csv.DictReader(open(test_file, 'rb'), delimiter=',', quotechar='"')
for line in csv_file:
host = line['url']
print cmd, host
pname = subprocess.Popen([cmd,host],stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = pname.communicate()
if pname.poll() == 100:
Here are the results. It executes the first url and then then the second in order.
September 5th, 2013, 07:36 AM
A threaded application is probably good enough. Another choice is multiprocessing. It's not completely simple and straight forward because the display phase (print statements) need to run single threaded to avoid jumbling the output. Offhand, you could use process synchronization; have each thread write to a separate file then concatenate the files after all threads have finished; use a separate thread to print results from a queue which the pinging threads test. I don't have time now to write code.
[/code] are essential for python code and Makefiles!