Support up-to-date botocore and urllib3 2.0

[jakob-keller]

Description of Change

This PR intends to improve general compatibility of aiobotocore within the Python ecosystem by relaxing the dependency specification of botocore, as well as boto3 and awscli.

Thanks to boto/botocore#3034, this PR also implicitly provides support for urllib3 2.0 for Python 3.10+. Installing urllib3>=2.0.0 causes unit tests to fail due to an outdated development dependency. This PR bumps docker to address the issue.

Assumptions

Upstream diff does not contain any changes that require adjustments to the aiobotocore codebase: Besides JSON data and docstring updates, there are only minor changes to botocore Python code and unit tests, none of which affect aiobotocore. This allows for the proposed widening of the botocore dependency specification.

Checklist for All Submissions

• I have added change info to CHANGES.rst
• If this is resolving an issue (needed so future developers can determine if change is still necessary and under what conditions) (can be provided via link to issue with these details): closes Be more permissive on boto3 version #1028, closes Bump botocore to latest version #1043, closes aiobotocore dependency on old version of botocore causes dependency conflicts aws/sagemaker-distribution#83, closes Feature Request: New Service support for "bedrock-runtime" terricain/aioboto3#317
  • Detailed description of issue
  • Alternative methods considered (if any)
  • How issue is being resolved
  • How issue can be reproduced
• If this is providing a new feature (can be provided via link to issue with these details):
  • Detailed description of new feature
  • Why needed
  • Alternatives methods considered (if any)

Checklist when updating botocore and/or aiohttp versions

• I have read and followed CONTRIBUTING.rst
• I have updated test_patches.py where/if appropriate (also check if no changes necessary)
• I have ensured that the awscli/boto3 versions match the updated botocore version

[codecov]

Codecov Report

Merging #1037 (f62c930) into master (5334722) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1037      +/-   ##
==========================================
- Coverage   86.27%   86.23%   -0.04%     
==========================================
  Files          58       58              
  Lines        5740     5740              
==========================================
- Hits         4952     4950       -2     
- Misses        788      790       +2     

Flag	Coverage Δ
unittests	86.23% <100.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tests/test_patches.py	95.91% <ø> (ø)
tests/test_version.py	95.28% <100.00%> (-1.89%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[JackKolb-ElationHealth]

I think this will help me with another issue. I'm trying to get a service definition for AWS bedrock supported across both aiobotocore and aioboto3. That requires a botocore bump to atleast 1.31.57.

[jakob-keller]

I think this will help me with another issue. I'm trying to get a service definition for AWS bedrock supported across both aiobotocore and aioboto3. That requires a botocore bump to atleast 1.31.57.

That should be possible once this PR is merged and a release is published.

[jakob-keller]

Until this PR is reviewed, I intend to monitor botocore releases and push updates to prevent the PR from becoming stale.

[just4brown]

Hi team, looks like we have a lot of folks blocked waiting for this change. Any maintainers available to review this? cc @thehesiod @terrycain @jettify

Thanks!

[jchacks]

Pinning to a patch version of botocore is very restrictive. Can the upper bound on 'botocore>=1.31.16,<1.31.63', be relaxed to <1.32.0? This is the same as boto3's upper bound on botocore found in https://github.com/boto/boto3/blob/develop/setup.py

[jakob-keller]

Pinning to a patch version of botocore is very restrictive.

Agreed.

Can the upper bound on 'botocore>=1.31.16,<1.31.63', be relaxed to <1.32.0?

Unfortunately, this is not possible due to the tight integration between botocore and aiobotocore.

[thehesiod]

looking! sorry for delay, have been super busy

[thehesiod]

my apologies for delay! kids/work/personal stuff has been honestly overwhelming this last month. Deploying shortly

[jakob-keller]

my apologies for delay! kids/work/personal stuff has been honestly overwhelming this last month. Deploying shortly

We hear you! 😄

[thehesiod]

FYI merging 2.7.0 + 2.7.1 in #1034 as 2.7.0 was never released

[jakob-keller]

FYI merging 2.7.0 + 2.7.1 in #1034 as 2.7.0 was never released

Sorry, my bad!

[paulmueller]

Sorry for posting on a closed PR. Is there any reason why botocore was pinned to

• botocore<1.31.65,>=1.31.16

instead of

• botocore<=1.31.65,>=1.31.16

The problem I am having is that boto3 1.28.65 requires botocore<1.32.0,>=1.31.65. I assume there is a breaking change in botocore 1.31.65?

So I am stuck with urllib3<2 and cannot use the latest versions of aiobotocore and s3fs.

[jakob-keller]

Sorry for posting on a closed PR. Is there any reason why botocore was pinned to

• botocore<1.31.65,>=1.31.16

instead of

• botocore<=1.31.65,>=1.31.16

botocore==1.31.65 was released after this PR was prepared and merged. Supporting botocore==1.31.65 would be trivial, but requires a separate PR and release.

You should be able to gain the benefits of this PR, if you work with boto3<=1.28.64.

[paulmueller]

Yes thanks, you are correct. The install works, albeit with a quite old version of s3fs (so I am waiting for s3fs to support aiobotocore 2.7.0).

$ pip install boto3==1.28.64 aiobotocore s3fs "urllib3>=2.0"
$ pip freeze
aiobotocore==2.7.0
aiohttp==3.8.6
aioitertools==0.11.0
aiosignal==1.3.1
async-timeout==4.0.3
attrs==23.1.0
boto3==1.28.64
botocore==1.31.64
charset-normalizer==3.3.0
frozenlist==1.4.0
fsspec==2021.7.0
idna==3.4
jmespath==1.0.1
multidict==6.0.4
python-dateutil==2.8.2
s3fs==2021.7.0
s3transfer==0.7.0
six==1.16.0
urllib3==2.0.7
wrapt==1.15.0
yarl==1.9.2

[potiuk]

Yes thanks, you are correct. The install works, albeit with a quite old version of s3fs (so I am waiting for s3fs to support aiobotocore 2.7.0).

Same here. Already merged in fsspec/s3fs#809

[potiuk]

And.... THANKS. This one will solve a lof of our problems in Apache Airflow ! We had to figure out a very weird scheme there to cope with pinning.