'Unclosed connection' warning when using S3 client

[maretodoric]

• Async AWS SDK for Python version: aioboto3==12.4.0
• Python version: 3.10.12
• Operating System: Ubuntu 22.04.2 LTS (Jammy Jellyfish)

Description

I'm receiving warning when using S3 client:

Unclosed connection
client_connection: Connection<ConnectionKey(host='bucket-name.s3.eu-central-1.amazonaws.com', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>

I have following function that is supposed to return information about object stored in S3:

async def get_s3_object(bucket: str, filepath: str):
    cfg = load_settings()
    session  = aioboto3.Session(aws_access_key_id=cfg.aws.key, aws_secret_access_key=cfg.aws.secret_key, region_name=cfg.aws.region)

    async with session.client('s3') as s3:
        try:
            s3_ob = await s3.get_object(Bucket=bucket, Key=filepath)
        except Exception as e:
            if type(e).__name__ == 'NoSuchKey':
                raise FileNotFoundError(f"File '{basename(filepath)}' not found in bucket: '{bucket}'.")
            else:
                raise
        if s3_ob['ResponseMetadata']['HTTPStatusCode'] != 200:
            raise Exception(f"HTTP Status Code invalid. Response: {s3_ob}")
        else:
            return s3_ob

Another function calls get_s3_object and everything works fine, but i get warning mentioned above.

I'm using the client as suggested, by creating a session then using the .client or .resource as context manager, but it appears as if it's not closing the S3 connection

[terricain]

Can you replicate this using aiobotocore alone? I have a suspicion its something to do with how the connection pool will keep connections open to speed up subsequent requests, whilst it's annoying I dont have much time to work on fixing it as it shouldn't affect operation.

[maretodoric]

Hm.. Good one.. It appears i can replicate using aiobotocore.

async def get_s3_object(bucket: str, filepath: str):
    filepath = str(filepath)
    cfg = load_settings()

    session = get_session()
    async with session.create_client('s3', region_name='eu-central-1',
                                     aws_secret_access_key=cfg.aws.secret_key,
                                     aws_access_key_id=cfg.aws.key) as client:
        # get object from s3
        try: s3_ob = await client.get_object(Bucket=bucket, Key=filepath)
        except Exception as e:
            if type(e).__name__ == 'NoSuchKey':
                raise FileNotFoundError(f"File '{basename(filepath)}' not found in bucket: '{bucket}'.")
            else:
                raise
        if s3_ob['ResponseMetadata']['HTTPStatusCode'] != 200:
            raise Exception(f"HTTP Status Code invalid. Response: {s3_ob}")
        else:
            return S3Object(bucket, filepath, s3_ob['ContentLength'], s3_ob['ContentType'])

The get_session is an import from aiobotocore.session

[terricain]

I'd suggest raising an issue over on aiobotocore as most likely the error is benign and should be silenced.

[maretodoric]

Yup, already did. Thanks a lot! :)

[pawelad]

FWIW, I'm also running into this problem:

async def stream_s3(
    s3_client: "S3Client",
    bucket_name: str,
    key: str,
) -> AsyncGenerator[bytes, None]:
    s3_obj = await s3_client.get_object(
        Bucket=bucket_name,
        Key=key,
    )

    async with s3_obj['Body'] as stream:
        body = await stream.read()
        yield body

    # This also doesn't work
    # async for chunk in s3_obj['Body']:
    #     yield chunk

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10956a360>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x109551730>, 604013.115)]']
connector: <aiohttp.connector.TCPConnector object at 0x1095698b0>

I'm using FastAPI and aioboto3.

Related (I think) issue:

• Unclosed response/connection errors aio-libs/aiobotocore#68

[pawelad]

FWIW, I managed to work around this by using AsyncExitStack, as shown in https://aioboto3.readthedocs.io/en/latest/usage.html#aiohttp-server-example