#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    7
    Rep Power
    0

    GetStatusOutput Function


    Hi

    I was using commands module function getstatusoutput to get the output of the command which is reading a very big file (size around 2 GB)

    When i run my python script, it get hanged when i call this statement
    command = '/opt/hadoop/bin/hadoop dfs -text <File_Path> 2>/dev/null'
    status, output = commands.getstatusoutput(command)

    it get hanged on the above statement.
    Could anyone help me in this knowing why it is happening.

    Thanks
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2012
    Location
    39N 104.28W
    Posts
    158
    Rep Power
    3
    Are you using Python 3? The commands module has been deprecated since 2.6 and removed in 3.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    7
    Rep Power
    0
    No its python 2.6
    and it is working fine for smal size file but it get get hanged for a big file.
  6. #4
  7. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,905
    Rep Power
    481
    You're quite sure the program isn't trying to read from stdin?

    Please print command immediately after you assign it.

    command = '/opt/hadoop/bin/hadoop dfs -text <File_Path> 2>/dev/null'
    print(command)


    Does it look like the command you'd use at the shell prompt?
    [code]Code tags[/code] are essential for python code and Makefiles!
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    7
    Rep Power
    0
    Hi

    i did print command just after the statement and it print as it is
    also it didnt ask for the input.

    And yes its the same command which i run on shell where it is running fine.

    Also when i run the same code using Oozie it get break over there with the exception

    *status, output = commands.getstatusoutput(command)
    *File "/usr/lib64/python2.4/commands.py", line 54, in getstatusoutput
    * *text = pipe.read()
    SystemError: Objects/stringobject.c:3565: bad argument to internal function
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    7
    Rep Power
    0
    Just to avoid confusion in the command statement
    command = '/opt/hadoop/bin/hadoop dfs -text <File_Path> 2>/dev/null'

    <File_Path> = i actually give a hdfs file path
  12. #7
  13. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,905
    Rep Power
    481
    Yes that was the point, since < > are shell redirection operators.
    "<File_Path> = i actually give a hdfs file path"

    Sorry, I didn't diagnose the problem.
    [code]Code tags[/code] are essential for python code and Makefiles!

IMN logo majestic logo threadwatch logo seochat tools logo