Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"unclosed transport" warning with long running client under Python 3.10 #1211

Open
6 tasks done
larashores opened this issue Oct 3, 2024 · 0 comments
Open
6 tasks done

Comments

@larashores
Copy link

larashores commented Oct 3, 2024

Describe the bug
Running Python 3.10, making any call to S3 without immediately closing the S3 client will cause the following warning to be emitted.

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x10814dcc0>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)

Here is a minimally reproducible example:

import asyncio
import aiobotocore
import warnings

import aiobotocore.session

warnings.simplefilter("default")  # Warning only goes to console if this call is made

async def main():
    session = aiobotocore.session.get_session(
        # Replace with real tokens
        {
            "AWS_ACCESS_KEY_ID": "test",
            "AWS_SECRET_ACCESS_KEY": "test",
            "AWS_SESSION_TOKEN": "test",
        }
    )
    async with session.create_client('s3') as s3:
        print("Entered client")
        try:
            await s3.get_object(Bucket='bucket', Key="key")
        except Exception as exc:
            print("Execption", exc)
        # On Python 3.10, a ResourceWarning will appear during the sleep. If instead the client is
        # closed immediately after the get_object() call is made, the warning will not appear.
        # However, this is not desirable, as we want to share the client across multiple coroutines.
        await asyncio.sleep(10)
    
    print("Exited client")
    await asyncio.sleep(10)

if __name__ == "__main__":
    asyncio.run(main())

Running this outputs the following:

Entered client
Execption An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x104d2dcc0>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exited client

Commenting out the first sleep, the warning disappears:

Entered client
Execption An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
Exited client

Checklist

  • I have reproduced in environment where pip check passes without errors
  • I have provided pip freeze results
  • I have provided sample code or detailed way to reproduce
  • I have tried the same code in botocore to ensure this is an aiobotocore specific issue
  • I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
  • I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.15.1
aiohappyeyeballs==2.4.3
aiohttp==3.10.8
aioitertools==0.12.0
aiosignal==1.3.1
async-timeout==4.0.3
attrs==24.2.0
botocore==1.35.23
frozenlist==1.4.1
idna==3.10
jmespath==1.0.1
multidict==6.1.0
python-dateutil==2.9.0.post0
six==1.16.0
typing_extensions==4.12.2
urllib3==2.2.3
wrapt==1.16.0
yarl==1.13.1

Environment:

  • Python Version: 3.10.11
  • OS name and version: MacOS Sonoma 14.6.1

Additional context

  • Running the code on Python 3.11 with the exact same dependencies will fix the issue.
  • I've tried similar code in botocore and the issue doesn't occur. Though the code isn't exactly equivalent since clients are not context managers in botocore
  • I've tried leaving long running sessions in aiohttp and the issue doesn't occur either.
  • I see the same behavior going all the way back into aiobotocore==1.4.0 and aiohttp==3.7.0. Any earlier versions and I start to see other errors that prevents the code from running.
@larashores larashores changed the title Unclosed transport error with long running client under Python 3.10 "unclosed transport" warning with long running client under Python 3.10 Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant