Incomplete docs for required endpoints

[The-Judge]

Description

We are using this module to connect to Azure DataLake and manage files within our DataLake resources with a corporate firewall in between. We need a definite list of IPs/DNS endpoints that are required to use this module and authenticate using personal and service principal based authentication to permit these on the firewall.
Authentification is delegated to our corporate Delegated AD auth service ([email protected]).

Available resources seem to be incomplete since the auth does not succeed even with all listed resources permitted on the firewall.

Error message:

 Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "extractor.py", line 516, in <module>
    main()
  File "extractor.py", line 486, in main
    upload_to_adls(
  File "extractor.py", line 377, in upload_to_adls
    adls_file_system_client = _create_adls_fs_client()
  File "extractor.py", line 368, in _create_adls_fs_client
    adl_creds = lib.auth(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/azure/datalake/store/lib.py", line 150, in auth
    out = get_token_internal()
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/azure/datalake/store/retry.py", line 93, in f_retry
    out = func(*args, **kwargs)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/azure/datalake/store/lib.py", line 143, in get_token_internal
    out = context.acquire_token_with_client_credentials(resource, client_id,
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/authentication_context.py", line 179, in acquire_token_with_client_credentials
    return self._acquire_token(token_func)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/authentication_context.py", line 128, in _acquire_token
    return token_func(self)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/authentication_context.py", line 177, in token_func
    return token_request.get_token_with_client_credentials(client_secret)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/token_request.py", line 310, in get_token_with_client_credentials
    token = self._oauth_get_token(oauth_parameters)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/token_request.py", line 112, in _oauth_get_token
    return client.get_token(oauth_parameters)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/adal/oauth2_client.py", line 263, in get_token
    resp = requests.post(token_url.geturl(),
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

---

Reproduction Steps

1. Put a restrictive firewall, which blocks all networking traffic, between the system that uses this module and the internet.
2. Try to create an authenticated ADLS FS Client using both authentication mechanisms (personal and service principal credentials):

    def _create_adls_fs_client(tenant_id, client_id=None, client_secret=None, azure_username=None, azure_password=None):
        if client_id is not None and client_secret is not None:
            # service principal auth
            adl_creds = lib.auth(
                client_id=client_id, client_secret=client_secret, tenant_id=tenant_id,
            )
        elif azure_username is not None and azure_password is not None:
            # active directory user auth
            adl_creds = lib.auth(
                username=azure_username, password=azure_password, tenant_id=tenant_id,
            )
        else:
            raise ValueError("Not all required auth parameters provided")
        return core.AzureDLFileSystem(adl_creds, store_name=azure_adls_account_name)

3. Apply all listed firewall rules on public docs to the firewall to permit these; you will find that the authentication does not succeed.

Environment summary

SDK Version: What version of the SDK are you using? (pip show azure-datalake-store)
Answer here: azure-datalake-store==0.0.49

Python Version: What Python version are you using? Is it 64-bit or 32-bit?
Answer here: 3.8.5, '64bit'

OS Version: What OS and version are you using?
Answer here: Debian GNU/Linux 10 (buster), MacOSX 10.15.6

Shell Type: What shell are you using? (e.g. bash, cmd.exe, Bash on Windows)
Answer here: bash on Linux, bash on MacOSX and none (directly launched Python script within a Docker container)

[akshat-harit]

This seems to be related to adal i.e. authentication.