Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Copy (managed transfer) should limit concurrency #349

Open
mjpieters opened this issue Aug 27, 2024 · 2 comments
Open

S3 Copy (managed transfer) should limit concurrency #349

mjpieters opened this issue Aug 27, 2024 · 2 comments
Assignees

Comments

@mjpieters
Copy link

When copying a large S3 object with s3.object.copy() / s3.client.copy(), aioboto3 will use multiple uploader() tasks to manage the transfer. However, it doesn't put any limit on the number of tasks created, ignoring the TransferConfig.max_request_concurrency value.

This should be easy to fix; just create a asyncio.Semaphore() and pass in the max_request_concurrency value for this transfer, then have the uploader() task start with async with <shared semaphore>: to limit the number of active concurrent tasks.

@terricain
Copy link
Owner

Ah yep, spot on, will fix at some point soon

@terricain terricain self-assigned this Aug 29, 2024
@terricain
Copy link
Owner

This is live in 13.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants
@mjpieters @terricain and others