#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0

    Need help understanding python's subprocess


    As a python hobiest have recently been experimenting with python's subprocess module and so far I can't quite seem to get a handle on it, so I thought I would pose a question that I think will help make things clearer. So far all of the examples I have seen basically set up a subprocess and then wait for that process to complete and then act on that information. But what I am trying to understand is in the case where you have two process and you want them both to run concurrently and act based on the output of the other. For example: Lets say you run as script called Parent.py which launches another script called Child.py as a subprocess. The child then starts it work collecting data or whatever. Meanwhile the parent has better things to do then wait around and continues on its way doing bigger and better things. Finally when Child.py is done messing around it sends a command to the Parent.py telling it to pay attention and run a print function stating that the child is finished. (Note: I understand this can be done using threading but for the sake of this exercise I would like to stick to using a subprocess). Is this possible, or even the correct way to use a subprocess?
  2. #2
  3. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,856
    Rep Power
    481

    Unix example


    Suppose b.sh is executable bash script
    Code:
    #! /bin/bash
    parent=$1
    while true
    do
      echo 'collecting data'
      sleep 4
      kill -10 $parent  # SIGUSR1
      if [[ 0 -ne $? ]]
      then
        break
      fi
    done
    and there is a program p.py
    Code:
    import os
    import time
    import signal
    import subprocess
    
    def handler(signal_number, stack_frame):
        print("Child home by midnight.  OK.")
    
    signal.signal(signal.SIGUSR1,handler)
    
    command = './b.sh %s &'%(os.getpid())
    
    child = subprocess.Popen(command, shell=True)
    print('Expected run time 20 seconds')
    for i in range(20):
        print('parent labors')
        time.sleep(1)
    and that these files are in the same directory on a unix system and also that my current directory is the same, then I can run this with the output shown:
    Code:
    $ python p.py
    Expected run time 20 seconds
    parent labors
    collecting data
    parent labors
    parent labors
    parent labors
    parent labors
    collecting data
    Child home by midnight.  OK.
    parent labors
    parent labors
    parent labors
    parent labors
    collecting data
    Child home by midnight.  OK.
    parent labors
    parent labors
    parent labors
    parent labors
    collecting data
    Child home by midnight.  OK.
    parent labors
    parent labors
    parent labors
    parent labors
    collecting data
    Child home by midnight.  OK.
    parent labors
    parent labors
    parent labors
    $ ./b.sh: line 8: kill: (12549) - No such process
    Note that it was the child's responsibility to run in the background (the ampersand in the command string). On windows system I'd use the start command, for example "start excel".
    [code]Code tags[/code] are essential for python code and Makefiles!
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0
    Thanks for the reply b49P23TIvg. I suddenly feel like I have jumped into the deep end of the pool. Although, I think now I am starting to see things a little clearer. I will mess around with the code you have provided and see where it takes me.
  6. #4
  7. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,856
    Rep Power
    481
    If you're unfamiliar with unix then both the command and the child program b.sh might be troublesome to understand.
    [code]Code tags[/code] are essential for python code and Makefiles!
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0
    Fortunately I am familiar with Unix and for the most part things are making sense. I understand the gist of what you posted. I just need to do some digging on the lines
    Code:
    def handler(signal_number, stack_frame):
    and 
    command = './b.sh %s &'%(os.getpid())
    I am not sure what "signal_number, stack_frame" are about and the command itself has me scratching my head but I'll get there eventually.
  10. #6
  11. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,856
    Rep Power
    481
    I looked in the python signal module documents which told me the signal handler requires two arguments.

    The signal number is easy---it ought to be 10. (You could use the same handler for different signals. In that case you'd know which signal was caught.)
    I don't know much about stack frames. These might depend on hardware and operating system software. (Guessing) The stack frame determines what code executes after the signal handler returns. If you knew all that stuff you could pop the stack to a lower level or change the destination. You could possibly implement long jump with this feature.

    handler gains control when the signal is sent. I didn't use the handler arguments, so I didn't care what they were named, and I could have defined as

    def handler(*args):


    Otherwise, the child process needed the process identification number of its parent. I passed this as an argument to b.sh .
    os.getpid()
    Last edited by b49P23TIvg; January 22nd, 2013 at 08:19 PM.
    [code]Code tags[/code] are essential for python code and Makefiles!
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0
    Right that makes sense I think I am starting to get it now. I also ran across an article Signal Receive notification of asynchronous system events that was pretty helpful. So between it and the link you posted it's making a lot more sense. I find that often times the key to figuring things out is knowing what to search for. You have definitely pointed me in the right direction. Thank you very much.
  14. #8
  15. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,856
    Rep Power
    481
    Back in the '80s I asked a similar question of one of the system operators. The answer was, as I recall, a pair of cooperating FORTRAN programs for DEC VAX-VMS. DEC had a great FORTRAN compiler and a huge library of system calls compatible with FORTRAN I was trying to understand at least at a conceptual level.
    [code]Code tags[/code] are essential for python code and Makefiles!
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0
    Wow, you have been at it a bit longer then I have. I am actually a Visual FX guy and I code tools for the software we use as kind of a hobby. I am currently trying to implement a pyside tool in a 3d program called Blender which has a built in python interpreter . While I have gotten pyside to work for the most part, it takes over the the main thread when the pyside Gui launches. Which is ok, just not super cool. It also seems as though I can't run pyside gui ouside the main thread so I was looking for a way around it. Using subprocessing looked like a possible solution but I was having trouble getting my mind around it. Which was why I was asking the question. Thanks again for all your help.
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    9
    Rep Power
    0
    Ok, now that I think I have a reasonable understanding of how to communicate using the handler. I was wondering if it's possible to pass some sort of data from the child to the parent as well, or vice versa. For example lets say that instead of printing "Child home by midnight" we let the child tells the parent when he leaves and later let the parent know when he gets home. I guess what I am trying to understand is how, if possible, to gather information from the child throughout it's life cycle then hand off that information to the parent to be acted on. I have run across quite a few examples where information is collected when the child process is finished, but can't seem to find any where the is any real communication between the parent and the child while both are active.
  20. #11
  21. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,856
    Rep Power
    481
    You could use a file or shared memory map to communicate. The signals would indicate "ready".

    This forum thread
    http://forums.devshed.com/python-programming-11/subprocess-then-read-then-write-to-same-shell-934972.html
    addresses the issue using a different technique. Note that one of the posts links to a still earlier same subject post. There's also some python module that helps interaction between parent and child. Maybe there's help in the standard distribution modules too.

    Use google to search these forums with

    site:forums.devshed.com insightful search terms
    Last edited by b49P23TIvg; January 23rd, 2013 at 05:07 PM.
    [code]Code tags[/code] are essential for python code and Makefiles!

IMN logo majestic logo threadwatch logo seochat tools logo