Problem with NTLM-authenticate proxy for HTTPS CONNECT

[railocius]

Hi,
I try to use HttpNtlmAuth to use for a NTLM proxy authentication request.

tested with Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] and also newest 2.7.14:

import requests
from requests_ntlm import HttpNtlmAuth
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

user = 'WORKGROUP\user' # or '.\user'
password = '1234'

http_proxy  = 'http://127.0.0.1:8080'
proxy_dict = {'http': http_proxy, 'https': http_proxy}

retries = Retry(total=10,
                read=5,
                connect=6,
                #these options don't seem to help
                #backoff_factor=1,
                #method_whitelist=(['HEAD', 'TRACE', 'GET', 'POST', 'CONNECT', 'OPTIONS', 'DELETE']), 
                #status_forcelist=[500, 502, 503, 504, 403, 407]
                )

session = requests.Session()
session.verify = False
session.mount('http://', HTTPAdapter(max_retries=retries))
session.mount('https://', HTTPAdapter(max_retries=retries))

session.proxies = proxy_dict
session.auth = HttpNtlmAuth(user, password)

#this works: HTTP GET
r = session.get('http://neverssl.com/')
print(r.text)

#HTTPS CONNECT fails!
r2 = session.get('https://www.google.com')
print(r2.text)

The last error message from the call stack is:

`ConnectionError: HTTPSConnectionPool(host='www.google.com', port=443): 
Max retries exceeded with url: /
 (Caused by ProxyError('Cannot connect to proxy.', 
error('Tunnel connection failed: 407 Proxy Authentication Required',)))`

I checked with Wireshark (and Npcap for loopback connection monitoring) and the HTTP connection looks like this:
HTTP GET:
Packet 1, request: GET http://neverssl.com/
Packet 2, response: HTTP/1.1 407 Proxy Authentication Required
Packet 3, request: GET http://neverssl.com/ HTTP/1.1 , Proxy-authorization: NTLM TlRMTV...
Packet 4, response: HTTP/1.1 407 Proxy Authentication Required, Proxy-Authenticate: NTLM TlRMT....
Packet 5, request: GET http://neverssl.com/ HTTP/1.1 Proxy-authorization: NTLM TlRMTVNT...(much longer than last time)
Packet 6, request: HTTP/1.1 200 OK

HTTPS CONNECT:
Packet 1, request: CONNECT www.google.com:443 HTTP/1.0 (the whole request is literally just this one line)
Packet 2, response: HTTP/1.1 407 Proxy Authentication Required, Proxy-Authenticate: NTLM
-> Error ConnectionError

If you want to try this yourself, you can set up a NTLM-auth proxy like this: https://stackoverflow.com/a/48238953/9043528

Ok, how can this be fixed ?
Is this a bug in HttpNtlmAuth that does not set the number of retries ?

[iyanmv]

Hi @railocius! I have opened an issue in urllib3 repo: urllib3/urllib3#1434

[YuMan-Tam]

I have exactly the same issue. The problem seems to have nothing to do with requests-ntlm. I believe, for https, somehow urllib3 (or requests) encounters the error before it uses the mechanism in requests-ntlm (I did print statements in the package but nothing was printed). This only happens in https but not http with the proxy.

[YuMan-Tam]

See my hack here if this is helpful: urllib3/urllib3#1434 (comment)

[dopstar]

see follow up comment: urllib3/urllib3#1434 (comment)

[chahilp]

I had exactly the same issue and though of making changes to the libraries, but it didn't allow me as the libraries are administrator controlled. But it worked for me the other way by downloading the cntlm from official repo https://sourceforge.net/projects/cntlm/files/cntlm/cntlm%200.92.3/ and configuring it as recommended in the proxy protected environment.