Opensearch IAM auth

[kizmanj]

Already posted about this in another ticket, but that issue has been closed since.

Tried using IAM auth to connect to OpenSearch from a container running in EKS, with
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Yente 4.1.0 seems to have updated components that helped, but did not fully resolve the problem.

Using these settings:
export set YENTE_INDEX_TYPE="opensearch"
export set YENTE_INDEX_URL="https://redacted/"
export set YENTE_OPENSEARCH_REGION="eu-central-1"
export set YENTE_OPENSEARCH_SERVICE="es"

The error currently looks like this:

Traceback (most recent call last):
File "/venv/bin/yente", line 33, in
sys.exit(load_entry_point('yente', 'console_scripts', 'yente')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/cli.py", line 44, in reindex
asyncio.run(update_index(force=force))
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/app/yente/search/indexer.py", line 188, in update_index
async with with_provider() as provider:
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/init.py", line 47, in with_provider
provider = await _create_provider()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/init.py", line 25, in _create_provider
return await OpenSearchProvider.create()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/opensearch.py", line 54, in create
await es.cluster.health(wait_for_status="yellow", timeout=5)
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/client/cluster.py", line 131, in health
return await self.transport.perform_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 375, in perform_request
await self._async_call()
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 198, in _async_call
await self._async_init()
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 163, in _async_init
self.set_connections(self.hosts)
File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 255, in set_connections
connections = list(zip(map(_create_connection, hosts), hosts))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 253, in _create_connection
return self.connection_class(metrics=self.metrics, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/http_aiohttp.py", line 149, in init
self.headers.update(urllib3.make_headers(basic_auth=http_auth))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/urllib3/util/request.py", line 121, in make_headers
] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}"
^^^^^^^^^^^^^^^^^
AttributeError: 'AWSV4SignerAuth' object has no attribute 'encode'

According to this discussion, adding these args to the opensearch provider here should help:

connection_class = RequestsHttpConnection

[pudo]

Thanks for doing all this research, Josef! The only challenge I have with the discussion in the ticket is that they're talking sync Python, and we're using async. There's a similar ticket here talking about async: opensearch-project/opensearch-py#698 - which introduces yet another class, AWSV4SignerAsyncAuth.

Maybe I can ask for your help on this: if I make that change in main, would you be able to run a specific docker hash to check if it works?

[pudo]

It's in this commit, building docker now: 1c26099

[pudo]

OK, tool a few rounds - docker pull ghcr.io/opensanctions/yente:sha-b18443a contains the fix

[kizmanj]

Thanks a lot, will try it later today!

[kizmanj]

tried it, but still getting the same error.

Will try a manual build with this later:
connection_class = RequestsHttpConnection

[pudo]

I'd be really surprised to see that work, since requests is a sync library. Can you share with me the updated stack trace from running this version, and I'll file a report on opensearch-py?

[kizmanj]

You are right, it did not help.

Here's the stack trace coming out of ghcr.io/opensanctions/yente:sha-b18443a image

Traceback (most recent call last):
File "/venv/bin/yente", line 33, in
sys.exit(load_entry_point('yente', 'console_scripts', 'yente')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/cli.py", line 44, in reindex
asyncio.run(update_index(force=force))
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/app/yente/search/indexer.py", line 188, in update_index
async with with_provider() as provider:
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/init.py", line 47, in with_provider
provider = await _create_provider()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/init.py", line 25, in _create_provider
return await OpenSearchProvider.create()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/yente/provider/opensearch.py", line 54, in create
await es.cluster.health(wait_for_status="yellow", timeout=5)
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/client/cluster.py", line 131, in health
return await self.transport.perform_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 375, in perform_request
await self._async_call()
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 198, in _async_call
await self._async_init()
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 163, in _async_init
self.set_connections(self.hosts)
File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 255, in set_connections
connections = list(zip(map(_create_connection, hosts), hosts))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 253, in _create_connection
return self.connection_class(metrics=self.metrics, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/opensearchpy/_async/http_aiohttp.py", line 149, in init
self.headers.update(urllib3.make_headers(basic_auth=http_auth))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/urllib3/util/request.py", line 121, in make_headers
] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}"
^^^^^^^^^^^^^^^^^
AttributeError: 'AWSV4SignerAsyncAuth' object has no attribute 'encode'

[pudo]

So I made this experiment but that won't even pass our tests (breaks opensearch): 196691d

I think it's just broken. Will need to file a ticket upstream.

[kizmanj]

I kept experimenting and got it working!

Added this line before the AWSV4SignerAsyncAuth line 44. (and it's import too)
kwargs["connection_class"] = AsyncHttpConnection

Idea came from here

[pudo]

I tried the same thing yesterday, but AsyncHttpConnection seems to have another bug: it doesn't properly translate HTTP error codes. If you look at the build log, it looks like all server-side errors (400 etc.) are being translated into HTTP 500 errors, which messes up all of our error handling downstream in the app.

[kizmanj]

I have seen this on my test deployment too.
It seems like it's trying to create a new temporary index with the same name as a previous one.
In my case i thought it was related to a previous interrupted reindex run. Removing the index from opensearch solved it.

[kizmanj]

Unfortunately can't say much about the 400->500 thing, as i'm not familiar with python or the codebase.