Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0

    Question httplib: closing a web connection properly


    My newest problem is I can't figure out how to close down my web connection properly. I wrote some code that's supposed to connect to an SSL enabled web server, pass some input, grab back the return data, parse through it and find the session id and hash and print it to a file.

    I've wrapped the app in a recursion function that I found in an book I read online.

    The problem in my code is that the connection never really closes (at least that's what I think is wrong) so I keep getting the exact sessionid and hash instead of getting a new one each time. If I restart the app manually and only have it run with 1 recursion the sessionid and hash change as expected.

    Any kick in the right direction would be appreciated. I thought using the h.close() would close the open web connection.

    Thanks!

    Code:
    # VERSION 2
    import httplib, urllib, linecache, string, os
    #Using input for value if raw input variable saved as string
    #could use raw_input and convert using int(x)
    x = input("Number of times to run? ")
    
    # FUNCTION TO REPEATE VIA RECURSION
    def recurse(x): 
        if x<=0: 
            # If x is less than or equal to 0 finish
            print "Done."
        else: 
            # Main working part of program
            # This section will connect to web site and get data
            params = urllib.urlencode({'username': 'test', 'password': 'test'})
            h = httplib.HTTPS(host = "test.site.com", port = 443, key_file = "fake.pem", cert_file = "fake.pem")
            h.putrequest('POST', '/scripts/cgi.exe?')
            h.putheader('Content-length', '%d'%len(params))
            h.putheader('Accept', 'text/plain')
            h.putheader('Host', 'test.site.com')
            h.endheaders()
            h.send(params)
            reply, msg, hdrs = h.getreply()
            data = h.getfile().read()
            file('testing.html', 'w').write(data)
            h.close()
            
            # Find exact line to match against for sessionID and user/pass
            test = linecache.getline('testing.html', 23)
            os.remove('testing.html')
            new = string.split(test, '.')
            print string.lstrip(new[6], chars = 'exe?'), new[7]
            file('sessionIDs.txt', 'a').write(string.lstrip(new[6], chars = 'exe?'))
            file('sessionIDs.txt', 'a').write('   ')
            file('sessionIDs.txt', 'a').write(string.lstrip(new[7]))
            file('sessionIDs.txt', 'a').write('\n')
            recurse(x-1)
    
    # MAIN BODY ;) Not much to it.  Check the else section of the recurse function
    recurse(x)
    EDIT: Had to try out Grim Archon's py2html.
    Last edited by rickt; March 8th, 2004 at 12:14 PM.
  2. #2
  3. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    I really can't see any problems here and to be honest it kinda has me a little comfused .

    As you probably know already sessions work by keeping track of a user via a unique ID; this ID shouldnt change each time you connect. But then i could be wrong here since.

    This doesnt attempt to do anything with the recived ID? You're simply connecting and storing session ID's?

    I'm just guessing but the python program is acting somewhat like a browser (when you close the browser the session is lost, otherwise it keeps going). This could also have to do with how the website works i.e. looking for the same IP or even to do with the HTTP itself. Just some thoughts

    What are you trying to do exactly?

    Mark
    programming language development: www.netytan.com Hula

  4. #3
  5. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0
    Originally Posted by netytan
    What are you trying to do exactly?
    I'm trying to make a file (sessionIDs.txt) that contains the sessionID and username/password+unknown hash. I want a couple hundred of them. There's a pattern for the userid/pass that I can visibly see and I'm trying to ascertain if there's another pattern in the user/pass+unknown data hash.

    I've located where in the returned data this information is located and I've extracted it using linecache and string.split and string.lstrip. My problem is I didn't understand how the httplib module worked. I thought when I used the close() thingy that it would have torn down the connection and when the script worked it's way back through a second time I would be establishing a new connection.

    I need to:
    Connect to a website
    Gather data from response
    Tear down connection
    Connect to same website as if from a new connection

    Perhaps httplib is too smart for this script? Is there another module or function that I should be using to perform this function?

    Thanks!
    -Rick
  6. #4
  7. Mini me.
    Devshed Novice (500 - 999 posts)

    Join Date
    Nov 2003
    Location
    Cambridge, UK
    Posts
    783
    Rep Power
    14
    Hi rickt,
    Out of interest, why are you trying to analyse the relationship? What do you plan to use the info for?
    To be honest, this does not sound like to sort of topic an open forum like this is used for or have I missed a point?

    Grim
  8. #5
  9. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0

    Post


    Originally Posted by Grim Archon
    Hi rickt,
    Out of interest, why are you trying to analyse the relationship? What do you plan to use the info for?
    To be honest, this does not sound like to sort of topic an open forum like this is used for or have I missed a point?

    Grim
    I'm working on a pen test. I think you have the point of what I'm trying to do and there is no nefarious purposes.

    Besides, by knowing my own sessionID I should not be able to brute force or guess someone elses sessionID. Or, if I can, the sessionID should be linked with another piece of information such as IP/port combination.

    The cgi app I'm testing doesn't perform the latter.

    Is this the kind of thing that isn't supposed to be discussed in this forum? I'll remove my script and try to initiate conversations over PM if that's more appropriate.

    -Rick
  10. #6
  11. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Just a guess. but you could try manually deleting 'h' after closing it although i dout this will have much effect its worth a try.

    I'm not really sure what other modules you could use for this since the two that spring to mind are built ontop of httplib...

    urllib  => http://www.python.org/doc/2.3.3/lib/module-urllib.html
    urllib2 => http://www.python.org/doc/2.3.3/lib/module-urllib2.html

    Another option open to you would be to write a program that connects and retrives a single session ID (writing it to a file) and call that program from within another program.

    The result, a file full of session IDs etc. However if you can do this i hope that you put it to good use!!!

    Mark.
    programming language development: www.netytan.com Hula

  12. #7
  13. Mini me.
    Devshed Novice (500 - 999 posts)

    Join Date
    Nov 2003
    Location
    Cambridge, UK
    Posts
    783
    Rep Power
    14
    Why use recursion, I can't see why it is needed and it could be causing your problems.

    I had a look at httplib.HTTPS - it has a lot of fun with reference counts to sockets. As an experiment I would do as Netytan suggests and explicitly delete the HTTPS object:
    Code:
    import httplib, urllib, linecache, string, os
    #Using input for value if raw input variable saved as string
    #could use raw_input and convert using int(x)
    
    def getdata(): 
        # Main working part of program
        # This section will connect to web site and get data
        params = urllib.urlencode({'username': 'test', 'password': 'test'})
        h = httplib.HTTPS(host = "192.168.11.4", port = 443, key_file = "fake.pem", cert_file = "fake.pem")
        print dir(h.sock)
        h.putrequest('POST', '/scripts/cgi.exe?')
        h.putheader('Content-length', '%d'%len(params))
        h.putheader('Accept', 'text/plain')
        h.putheader('Host', 'test.site.com')
        h.endheaders()
        h.send(params)
        reply, msg, hdrs = h.getreply()
        data = h.getfile().read()
        file('testing.html', 'w').write(data)
        h.close()
        del h
        
        # Find exact line to match against for sessionID and user/pass
        test = linecache.getline('testing.html', 23)
        os.remove('testing.html')
        new = string.split(test, '.')
        print string.lstrip(new[6], chars = 'exe?'), new[7]
        file('sessionIDs.txt', 'a').write(string.lstrip(new[6], chars = 'exe?'))
        file('sessionIDs.txt', 'a').write('   ')
        file('sessionIDs.txt', 'a').write(string.lstrip(new[7]))
        file('sessionIDs.txt', 'a').write('\n')
    
    
    
    x = input("Number of times to run? ")
    while x: 
        getdata()
        x -= 1
    If that does not work then try a batch file to call the python code n times. At least you can expect it to tidy up then

    BTW - py2htm version 0.61 includes a switch to stop messing with entities like < and > when producing forum code.
  14. #8
  15. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0
    Originally Posted by Grim Archon
    Why use recursion, I can't see why it is needed and it could be causing your problems.
    I just read about it, so I thought I'd try it.

    Originally Posted by Grim Archon
    Code:
        print dir(h.sock)
    What's this line supposed to do? Running it just yeilds an error.

    Traceback (most recent call last):
    File "C:/Python23/test.py", line 38, in -toplevel-
    webConnection()
    File "C:/Python23/test.py", line 10, in webConnection
    print dir(h.sock)
    AttributeError: HTTPS instance has no attribute 'sock'

    So I'm guessing that sock isn't what you were looking for, but I'm not sure.

    EDIT: Incidentally, using the del h didn't work. I wrote a 4 line while looping script to call the original script. That works.

    I'd still like to know why the close() doesn't close the entire session.

    Can someone tell me if this section of httplib.py is what's causing my problem?

    Code:
        def close(self): 
            self._conn.close()
    
            # note that self.file == response.fp, which gets closed by the
            # superclass. just clear the object ref here.
            ### hmm. messy. if status==-1, then self.file is owned by us.
            ### well... we aren't explicitly closing, but losing this ref will
            ### do it
            self.file = None
    Last edited by rickt; March 9th, 2004 at 12:11 PM.
  16. #9
  17. Mini me.
    Devshed Novice (500 - 999 posts)

    Join Date
    Nov 2003
    Location
    Cambridge, UK
    Posts
    783
    Rep Power
    14
    Left a little exploratory code in by mistake.
    When you say calling the other script I guess you meant os.spawnX or os.popen?

    For me, the flow diagram at the top of the httplib module underlines the complexity of the protocol. It looks like the http protocol(s) invite difficult constructions. If it was easy I guess we would all be doing it
  18. #10
  19. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Like Grim said, HTTP isn't simple. By its very nature it is HTTP is used to connectiong and disconnecting, which i guess is the reason behind this problem.

    A web browser for instance is constantly doing this, but the session ID change each time. Or at least not in my experiance.

    Just a guess, but still, now you know

    Mark.
    programming language development: www.netytan.com Hula

  20. #11
  21. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0

    Post


    Grim: I used os.open(python\path python\script\path) enclosed in a while loop. <shrug> (no shrugging smiley ) I'll have to look up os.spawn and os.popen. I just found os.open first and it worked so I stopped looking after that.

    netytan: I'm just disappointed that the httplib doesn't tear down the connection properly when a close is used. I'm pretty sure that because the connection isn't *explicitly* closing (as indicated in the comments for httplib.py that I pointed out) it's being kept open.

    I'm going to research a bit more.
  22. #12
  23. Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2004
    Posts
    23
    Rep Power
    0
    Ok, I performed a small piece of research. It appears that the fin ack is not sent when using HTTPS but it is sent when using HTTP.

    So, is there a way to contact the creator of the httplib.py library? Or is there a place to check and see if this is a known issue?

    Thanks!

    -Rick
  24. #13
  25. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    My bad sorry, i musta missed that bright red comment at the end of your post . Although the statments in my last post still apply; HTTP is used to of thing. Whether or not that is the reason behind me i imagine it must be playing some part in this. You mean os.popen() right, os.open() is the same as the __builtin__ file() and open() functions.

    You could try editing the httplib source to fix the problem however i wouldnt sugest it . Let us know what you find out.

    Mark.
    Last edited by netytan; March 9th, 2004 at 05:42 PM.
    programming language development: www.netytan.com Hula

  26. #14
  27. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    If this is a bug in the library then you should probably look at the Python bug board on source forge.

    http://sourceforge.net/tracker/?grou...70&atid=105470

    These guys work way way too hard . But they should be able to tell you everything you need to know - if this is intended or just a bug in httplib. Then, assuming this is a bug it will be adressed and hopefully fixed for Python 2.3.4 .

    Mark.
    programming language development: www.netytan.com Hula

  28. #15
  29. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Feb 2003
    Location
    Canada
    Posts
    543
    Rep Power
    24
    You could try copying all of the htmlib source, pasting it in a text editor, making the changes you wanted, and saving it to exactly where htmlib is, only call this htmlib2.
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo