S3 upload_fileobj tasks finally fail with SignatureDoesNotMatch

[Kakadus]

• Async AWS SDK for Python version: 13.2.0
• Python version: 3.12
• Operating System: Linux

Description

I'm creating many upload_fileobj tasks to upload in the "background". Sometimes, I observe that tasks fail with a SignatureDoesNotMatch exception. After some research I found this issue which might be related.

After reading the issue, I guess that wrapping upload_fileobj in a semaphore may resolve the issue.

What I Did

Sadly, I don't have simple code to reproduce the issue consistently.

Task failed" task="<Task finished name='Task-14963' coro=<collection_task.<locals>.store_data() done, defined at ...> exception=ClientError('An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.')>" exc_info="Traceback (most recent call last):
  File "...", line 88, in store_data
    await s3.upload_fileobj(BytesIO(byte_value), BUCKET, str(key))
  File "/app/.venv/lib/python3.12/site-packages/aioboto3/s3/inject.py", line 374, in upload_fileobj
    await self.put_object(
  File "/app/.venv/lib/python3.12/site-packages/aiobotocore/client.py", line 412, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

[terricain]

Can you test this again using aiobotocore and put_object instead

[Kakadus]

Okay, using a local s3 (minio), I were able to cause a RequestTimeTooSkewed error after ~8000 requests with aiobotocore using this code:

import asyncio

from aiobotocore.session import get_session
from itertools import repeat
from types_aiobotocore_s3.client import S3Client

AWS_ACCESS_KEY_ID = "minioadmin"
AWS_SECRET_ACCESS_KEY = "minioadmin"


async def put_obj(client: S3Client, i, data):
    await asyncio.sleep(1)
    print("started", i)
    response = await client.put_object(Bucket="test", Key="test", Body=data)
    print("finished", i)
    return response


async def inference_using_gather(data):
    session = get_session()
    async with session.create_client(
        "s3",
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        endpoint_url="http://localhost:9000",
    ) as s3_client:
        a = [asyncio.create_task(put_obj(s3_client, i, b)) for i, b in enumerate(data)]
        await asyncio.gather(*a)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    data = repeat("1" * 10000, 100000)
    prediction_dict = loop.run_until_complete(inference_using_gather(data))

Traceback:

...
finished 7629
finished 7639
Traceback (most recent call last):
  File ".../test/put.py", line 38, in <module>
    prediction_dict = loop.run_until_complete(inference_using_gather(data))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File ".../test/put.py", line 32, in inference_using_gather
    await asyncio.gather(*a)
  File ".../test/put.py", line 14, in put_obj
    response = await client.put_object(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/aiobotocore/client.py", line 412, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (RequestTimeTooSkewed) when calling the PutObject operation: The difference between the request time and the server's time is too large.

[terricain]

Ok so if you still hit issues with aiobotocore, you'll want to raise the issue there, as that library does most of the legwork that botocore would for boto3.

That being said, I think you're most likely flooding the event loop with tasks, so you have 100000 put's all competing to write, all started, but by the time they've gotten to the point of writing the signatures have expired. You'd be better off starting say 4 co-routines, all waiting to pop items from the same async queue, and feed the queue with data, that way you only have 4 tasks trying to put objects in parallel