#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0

    Parse Error when using showforms() command with Twill


    Hi, bare with me with this as it's my first python program. I've spent a few hours Googling and testing for a solution but I'm at that point where I think I just need a helping hand from one of you kind people.

    My idea for this program is to have a simple (supposedly) script that monitors what time it is, and when it's in a certain time range (for example 6am to 7pm) it navigates to opendns.com and blocks certain websites using the web content filtering feature.

    Now because I'm quite the novice when it comes to python I thought I'd start it simple and just figure out the commands to login to the website and block a website and worry about the monitoring of the time etc later on. But sadly I'm having trouble with that aswell.

    I'm using http://twill.idyll.org/ but not sure if that's a good idea or not. It's the only one I could find apart from mechanize (Which I couldn't find the right documentation for, but perhaps I'm just not looking in the right places)
    Here is my code (well, it's not really code yet. Just a list of commands for the Python Shell.):

    Code:
    from twill import get_browser
    from twill.commands import *
    
    username = "username@email.com" # email for opendns
    password = "thisisthepassword" # password for opendns
    b = get_browser()
    
    b.go("https://dashboard.opendns.com/")
    b.showforms()
    
    fv("2", "username", username)
    fv("2", "password", password)
    showforms()
    
    submit("sign-in")
    
    b.showforms()
    
    b.go ("https://dashboard.opendns.com/settings/*MYNETWORKID*/content_filtering") # I replaced my network ID due to privacy reasons but this is basically the URL to the web content filtering page on OpenDNS for a network
    
    b.showforms()

    Now that's where my problem starts. On that last b.showforms() I get an error:

    Code:
    Traceback (most recent call last):
      File "<pyshell#43>", line 1, in <module>
        b.showforms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\browser.py", line 225, in showforms
        forms = self.get_all_forms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\browser.py", line 259, in get_all_forms
        global_form = self._browser.global_form()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_mechanize.py", line 446, in global_form
        return self._factory.global_form
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\utils.py", line 334, in get_global_form
        return self.factory.global_form
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 521, in __getattr__
        self.forms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 534, in forms
        self._forms_factory.forms())
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 226, in forms
        raise ParseError(exc)
    ParseError: <unprintable ParseError object>
    Could anyone help me out or point me in the right direction? If you think I'm going about it completely wrong please do tell me, even if you think I should use another language. Like I said this is my first program so I'm not overly attached to Python, I just had been told it was perfect for a program like this.

    Sorry If I posted this question in the wrong format or something. I tried my best to adhere to the guidelines, if there's anything I need to change or more information I need to provide just let me know.
    Thanks in advance for anyone who takes the time to help me, I know this probably seems like a question with an obvious answer to most of you.
  2. #2
  3. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,886
    Rep Power
    481

    Guessing


    It's an html error. Can you force twill to overcome bad data? See strict=False:
    [quote=python documents]
    20.2. html.parser Simple HTML and XHTML parser
    Source code: Lib/html/parser.py

    This module defines a class HTMLParser which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.

    class html.parser.HTMLParser(strict=False)[/code]
    [code]Code tags[/code] are essential for python code and Makefiles!
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0
    Hmmm, what would be the correct way to go about doing that?

    I found on http://twill.idyll.org/browsing.html you can apparently do this using
    Code:
    config allow_parse_errors 1
    but that does not work for me in the python shell. It instead returns
    Code:
    SyntaxError: invalid syntax
    so I tried to make it work by doing
    Code:
    config [allow_parse_errors] [1]
    but that just returns:
    Code:
    Traceback (most recent call last):
      File "<pyshell#16>", line 1, in <module>
        config [allow_parse_errors] [1]
    NameError: name 'allow_parse_errors' is not defined
    And the same result happens if I try
    Code:
    config (allow_parse_errors, "1")
    What am I doing wrong?

    Thanks for the help by the way.
  6. #4
  7. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,886
    Rep Power
    481
    config['allow_parse_errors'] = 1

    Assuming config is a dictionary, and that the string
    'allow_parse_errors' is in some way a useful key.

    Printing config and presenting result to the forum might be enlightening.

    print(config)
    [code]Code tags[/code] are essential for python code and Makefiles!
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0
    Originally Posted by b49P23TIvg
    config['allow_parse_errors'] = 1

    Assuming config is a dictionary, and that the string
    'allow_parse_errors' is in some way a useful key.

    Printing config and presenting result to the forum might be enlightening.

    print(config)
    I'm getting a different error now:

    Code:
    Traceback (most recent call last):
      File "<pyshell#17>", line 1, in <module>
        config['allow_parse_errors'] = "1"
    TypeError: 'function' object does not support item assignment
    When I try to print the config I just get
    Code:
    <function config at 0x02A9B1B0>
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    22
    Rep Power
    0
    EDIT: whoops sorry
  12. #7
  13. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,886
    Rep Power
    481
    Now we know that config is a function.

    Try

    config(allow_parse_errors = 1)


    It's really really important to know datatype.

    print(type(config))

    would have helped.
    [code]Code tags[/code] are essential for python code and Makefiles!
  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0
    Originally Posted by b49P23TIvg
    Now we know that config is a function.

    Try

    config(allow_parse_errors = 1)


    It's really really important to know datatype.

    print(type(config))

    would have helped.
    Here's the error I'm getting now

    Code:
    >>> config(allow_parse_errors = 1)
    
    Traceback (most recent call last):
      File "<pyshell#3>", line 1, in <module>
        config(allow_parse_errors = 1)
    TypeError: config() got an unexpected keyword argument 'allow_parse_errors'
    
    >>> print(type(config))
    <type 'function'>
    Thanks for your help by the way.
  16. #9
  17. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,886
    Rep Power
    481
    Great, now try these experiments

    >>> help(config)


    >>> help(config.__class__)



    or you could read the twill instructions
    [code]Code tags[/code] are essential for python code and Makefiles!
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0
    Originally Posted by b49P23TIvg
    Great, now try these experiments

    >>> help(config)


    >>> help(config.__class__)



    or you could read the twill instructions
    Hmm, it appears they've removed the allow parse errors function

    Code:
    >>> help(config)
    Help on function config in module twill.commands:
    
    config(key=None, value=None)
        >> config [<key> [<int value>]]
        
        Configure/report various options.  If no <value> is given, report
        the current key value; if no <key> given, report current settings.
        
        So far:
        
         * 'acknowledge_equiv_refresh', default 1 -- follow HTTP-EQUIV=REFRESH
         * 'readonly_controls_writeable', default 0 -- make ro controls writeable
         * 'require_tidy', default 0 -- *require* that tidy be installed
         * 'use_BeautifulSoup', default 1 -- use the BeautifulSoup parser
         * 'use_tidy', default 1 -- use tidy, if it's installed
         * 'with_default_realm', default 0 -- use a default realm for HTTP AUTH
        
        Deprecated:
         * 'allow_parse_errors' has been removed.
    Here's what the other one returned:
    Code:
    >>> help(config.__class__)
    Help on class function in module __builtin__:
    
    class function(object)
     |  function(code, globals[, name[, argdefs[, closure]]])
     |  
     |  Create a function object from a code object and a dictionary.
     |  The optional name string overrides the name from the code object.
     |  The optional argdefs tuple specifies the default argument values.
     |  The optional closure tuple supplies the bindings for free variables.
     |  
     |  Methods defined here:
     |  
     |  __call__(...)
     |      x.__call__(...) <==> x(...)
     |  
     |  __delattr__(...)
     |      x.__delattr__('name') <==> del x.name
     |  
     |  __get__(...)
     |      descr.__get__(obj[, type]) -> value
     |  
     |  __getattribute__(...)
     |      x.__getattribute__('name') <==> x.name
     |  
     |  __repr__(...)
     |      x.__repr__() <==> repr(x)
     |  
     |  __setattr__(...)
     |      x.__setattr__('name', value) <==> x.name = value
     |  
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |  
     |  __closure__
     |  
     |  __code__
     |  
     |  __defaults__
     |  
     |  __dict__
     |  
     |  __globals__
     |  
     |  func_closure
     |  
     |  func_code
     |  
     |  func_defaults
     |  
     |  func_dict
     |  
     |  func_doc
     |  
     |  func_globals
     |  
     |  func_name
     |  
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |  
     |  __new__ = <built-in method __new__ of type object>
     |      T.__new__(S, ...) -> a new object with type S, a subtype of T

    From this it looks like I would have to just use BeautifulSoup or Tidy or a combination. But I have them both installed already.

    When I try something like:

    Code:
    config(use_tidy = 1)
    I get back:

    Code:
    Traceback (most recent call last):
      File "<pyshell#4>", line 1, in <module>
        config(use_tidy = 1)
    TypeError: config() got an unexpected keyword argument 'use_tidy'
    But isn't that the keyword it says to use in the config?
  20. #11
  21. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,886
    Rep Power
    481
    Code:
    >>> import  twill.commands as tc
    >>> tc.config(key='require_tidy',value=True)
    >>> tc.config(key='use_tidy',value=1)
    >>> tc.config(key='require_BeautifulSoup',value=False)
    >>> tc.config(key='use_BeautifulSoup',value=False)
    >>> tc.config()
    current configuration:
    	acknowledge_equiv_refresh : True
    	allow_parse_errors : True
    	readonly_controls_writeable : False
    	require_BeautifulSoup : False
    	require_tidy : True
    	use_BeautifulSoup : False
    	use_tidy : True
    	with_default_realm : False
    
    >>> tc.config('acknowledge_equiv_refresh',True) # you can omit key= and value=
    >>> 
    >>>
    [code]Code tags[/code] are essential for python code and Makefiles!
  22. #12
  23. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    6
    Rep Power
    0
    Originally Posted by b49P23TIvg
    Code:
    >>> import  twill.commands as tc
    >>> tc.config(key='require_tidy',value=True)
    >>> tc.config(key='use_tidy',value=1)
    >>> tc.config(key='require_BeautifulSoup',value=False)
    >>> tc.config(key='use_BeautifulSoup',value=False)
    >>> tc.config()
    current configuration:
    	acknowledge_equiv_refresh : True
    	allow_parse_errors : True
    	readonly_controls_writeable : False
    	require_BeautifulSoup : False
    	require_tidy : True
    	use_BeautifulSoup : False
    	use_tidy : True
    	with_default_realm : False
    
    >>> tc.config('acknowledge_equiv_refresh',True) # you can omit key= and value=
    >>> 
    >>>
    Hmm, I'm getting

    Code:
    Traceback (most recent call last):
      File "<pyshell#21>", line 1, in <module>
        showforms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\commands.py", line 371, in showforms
        browser.showforms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\browser.py", line 225, in showforms
        forms = self.get_all_forms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\browser.py", line 259, in get_all_forms
        global_form = self._browser.global_form()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_mechanize.py", line 446, in global_form
        return self._factory.global_form
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\utils.py", line 334, in get_global_form
        return self.factory.global_form
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 521, in __getattr__
        self.forms()
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 534, in forms
        self._forms_factory.forms())
      File "C:\Python27\lib\site-packages\twill-0.9-py2.7.egg\twill\other_packages\_mechanize_dist\_html.py", line 226, in forms
        raise ParseError(exc)
    ParseError: <unprintable ParseError object>
    Still,

    I'm starting to wonder, perhaps it's just not possible with Twill? I mean it is quite an old module I believe. Is it just not compatible with the new(ish) website Opendns?

IMN logo majestic logo threadwatch logo seochat tools logo