Problem with urllib2

Hi all,
I am facing authentication errors while using urllib2. Pasted Below is the code i wrote, followed by the errors thrown. Could someone help me in this regard.

Python version:
D:\Data\Python Scripts\review>python
ActivePython 2.7.2.5 (ActiveState Software Inc.) based on
Python 2.7.2 (default, Jun 24 2011, 12:22:14) [MSC v.1500 64 bit (AMD64)] on win32

Code:
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password. If we knew the realm, we could use it instead of ``None``.
password_mgr.add_password(None, top_level_url, 'user', 'password')
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
#--handler = urllib2.HTTPDigestAuthHandler(password_mgr)
print "A: ", proxy_support
# create "opener" (OpenerDirector instance)
opener = urllib2.build_opener(handler)
print "B: ", opener
# Install the opener.
# Now all calls to urllib2.urlopen use our opener.
urllib2.install_opener(opener)
# use the opener to fetch a URL
obj = opener.open(top_level_url)



Errors:


D:\Data\Python Scripts\review>shpoint.py
A: <urllib2.ProxyHandler instance at 0x00000000022D9448>
B: <urllib2.OpenerDirector instance at 0x00000000022D91C8>
Traceback (most recent call last):
File "D:\Data\Python Scripts\review\shpoint.py", line 37, in <module>
obj = getHTTPobject(mysite)
File "D:\Data\Python Scripts\review\shpoint.py", line 30, in getHTTPobject
obj = opener.open(top_level_url)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 859, in http_error_auth_reqed
response = self.retry_http_basic_auth(host, req, realm)
File "C:\ActivePython27\lib\urllib2.py", line 872, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "C:\ActivePython27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\ActivePython27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ActivePython27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\ActivePython27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\ActivePython27\lib\urllib2.py", line 884, in http_error_401
url, req, headers)
File "C:\ActivePython27\lib\urllib2.py", line 850, in http_error_auth_reqed
headers, None)
urllib2.HTTPError: HTTP Error 401: basic auth failed

Do you know what type of authentication the server you are trying to access is using? Failing that what web server is it?

I am trying to login to a Microsoft sharepoint server in our office, and I am not sure about the authentication method. Is there a way to find out in windows.

Regards
Sachin

You should ask your sys administrator that question, after all he/she installed it. We use LDAP authentication for ours at work and I think the default authentication method is Windows Authentication. From your output of the urllib2 errors, looks like you're doing basic http authentication, which is also a completely different authentication protocol.

Microsoft normally use NTLM (NT LAN Manager) authentication although I don't have a Sharepoint server to confirm this.

If you have pywin32 have a look in Lib\site-packages\win32\Demos\security\sspi. I think fetchurl would be the one to start with; it uses httplib rather than urllib2 but handles the authentication side.

Sorry for the late reply, but now I have something working. The authentication type used by Sharepoint as you correctly pointed out is "NTLM", so I traced a "python-ntlm" module (ver 1.0.1)

and Here is what I did:
1. installed python-ntlm module
2. Added the following workaround in "HTTPNtlmAuthHandler.py" (found on code.google)
auth_header_value = r.getheader(auth_header_field, None)
+ if ',' in auth_header_value:
+ auth_header_value, postfix = auth_header_value.split(',', 1)

(ServerChallenge, NegotiateFlags) = ntlm.parse_NTLM_CHALLENGE_MESSAGE(auth_header_value[5:])
3. used the WindowsHttpAuthenticated module, and the following code works.

#-- Code Begins ---
import urllib2
from ntlm import HTTPNtlmAuthHandler

user = 'domain-name\\user'
password = "password"
url = mysite

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)

# create the NTLM authentication handler
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)

# create and install the opener
opener = urllib2.build_opener(auth_NTLM)
urllib2.install_opener(opener)

# retrieve the result
response = urllib2.urlopen(url)
print(response.read())
#-- Code Ends ---

Now, I don't see any errors and have some output. Next, I will try to use the SUDS module to work with Sharepoint. (As of now getting some "xml.sax._exceptions.SAXParseException" errors) But for that I will try a little more before posting a new question later.

Warm Regards
Sachin