#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    5
    Rep Power
    0

    Python and Mechanize for web retrieval


    Hi;

    I am trying to learn about using Mechanize to post a form on a website, then to retrieve the page that follows the post.

    I have done a web search for examples, but they are to complex for entry level.

    My code is not accepting any of the following
    br['saledate'] = "12/11/2012"
    br.form['saledate'] = "12/11/2012"
    br.forms['saledate'] = "12/11/2012"

    I get the following error...
    Traceback (most recent call last):
    File "C:\Python27\jcommlist3.py", line 11, in <module>
    br.forms['saledate'] = "12/11/2012"
    TypeError: 'instancemethod' object does not support item assignment

    Here is the code so far;

    br = Browser()
    br.open(LOGIN_URL)
    br.select_form(nr=0)
    br.forms['saledate'] = "12/11/2012"
    resp = br.submit
    print resp

    for f in br.forms():
    print f

    it runs if I take out br.forms['saledate'] = "12/11/2012"
    and gives me back the following
    (as a new user I am not allowed to post urls, so I xxx'd that out...)

    <POST xxxxxxxxx/content.asp?page_id=9 application/x-www-form-urlencoded
    <HiddenControl(flag=1) (readonly)>
    <TextControl(caseno=)>
    <TextControl(plaintiff=)>
    <TextControl(defendant=)>
    <TextControl(address=)>
    <TextControl(city=)>
    <TextControl(zip=)>
    <TextControl(attorney=)>
    <TextControl(saledate=)>
    <SubmitControl(<None>=Begin Search) (readonly)>>

    any suggestions or example scripts would be really appreciated.
  2. #2
  3. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2005
    Location
    U.S A.Z.
    Posts
    24
    Rep Power
    0
    Hi I have just recently looked into using the mechanize module so I am no expert but instead of
    Code:
    br.forms['saledate'] = "12/11/2012"
    try
    Code:
     br.form['saledate'] = "12/11/2012"
    As in br.form[] Let me know if this works for you.
    Last edited by knutrainer; December 9th, 2012 at 10:43 PM. Reason: wrong code
  4. #3
  5. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Location
    Woodland Hills, Los Angeles County, California, USA
    Posts
    9,625
    Rep Power
    4247
    Do you have to use Mechanize? Is there a chance your program could run in a Windows environment. If so, I wrote a module a while back which controls an Internet Explorer instance and works better than Mechanize in some cases (e.g. if the site has javascript controls). It also comes with complete examples. Let me know if you want it.
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    5
    Rep Power
    0
    Originally Posted by Scorpions4ever
    Do you have to use Mechanize? Is there a chance your program could run in a Windows environment. If so, I wrote a module a while back which controls an Internet Explorer instance and works better than Mechanize in some cases (e.g. if the site has javascript controls). It also comes with complete examples. Let me know if you want it.
    Hi Scorpions4ever,
    I would be interested in the scraper, please pm me if you can sent it to me. As a new member, I can't pm you yet.
    Thanks in advance.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    5
    Rep Power
    0
    Originally Posted by knutrainer
    Hi I have just recently looked into using the mechanize module so I am no expert but instead of
    Code:
    br.forms['saledate'] = "12/11/2012"
    try
    Code:
     br.form['saledate'] = "12/11/2012"
    As in br.form[] Let me know if this works for you.
    Hi;
    br.form was correct. I also had the wrong form number,

    br.select_form(nr = 1)

    got it to work correctly.
  10. #6
  11. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Location
    Woodland Hills, Los Angeles County, California, USA
    Posts
    9,625
    Rep Power
    4247
    Originally Posted by jetus
    Hi Scorpions4ever,
    I would be interested in the scraper, please pm me if you can sent it to me. As a new member, I can't pm you yet.
    Thanks in advance.
    Here you go:
    Download link
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    5
    Rep Power
    0
    Hi Mayuk;

    Do you know any of the Bose family that live in Toronto? I backpacked Europe for 4 months with Anindo Bose..
    will have to send you a pic, lol!

    Really like what you did with IEC, great work! I can think of a dozen uses already.

    In running the example code..
    import IEC
    ie = IEC.IEController()
    ie.Navigate("new users cant post url's")
    ie.SetInputValue('q', 'mayukh bose')
    ie.ClickButton(caption='Google Search')

    I get this message...

    Traceback (most recent call last):
    File "C:/Python27/code/iec/excode.py", line 5, in <module>
    ie.ClickButton(caption='Google Search')
    File "C:\Python27\IEC.py", line 141, in ClickButton
    raise 'Button ' + itemtocheck + ' was not found.'
    TypeError: exceptions must be old-style classes or derived from BaseException, not str

    any explanation would be helpful, as I am a noobie

IMN logo majestic logo threadwatch logo seochat tools logo