Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Events: Upload all images on minIO S3 #39

Merged
merged 6 commits into from
Feb 16, 2024

Conversation

Mulugruntz
Copy link
Contributor

@Mulugruntz Mulugruntz commented Feb 10, 2024

Click to see python code to list content

from dvalin_tools.lib.s3 import S3Client
from dvalin_tools.lib.settings import DvalinSettings


def main():
    bucket_name = "event"
    s3_client = S3Client(settings=DvalinSettings())
    ls_s3 = s3_client.list_subdir_details(bucket_name=bucket_name)
    total_count, total_size = 0, 0
    for subdir, cnt, size in ls_s3:
        human_readable_size = size / 1024 / 1024
        print(f"{s3_client.s3_endpoint}/{bucket_name}/{subdir}: {cnt:>5} files, {size:>12} bytes ({human_readable_size:>8.2f} MB)")
        total_count += cnt
        total_size += size

    print(f"{'='*86}")
    print(f"Total      {s3_client.s3_endpoint}/{bucket_name}: {total_count:>5} files, {total_size:>12} bytes ({total_size / 1024 / 1024:>8.2f} MB)")
    print(" " * 73 + f"( ={total_size / 1024 / 1024 / 1024:>6.2f} GB)")


if __name__ == "__main__":
    main()

https://s3-api.dval.in/event/de-de/2021:   518 files,    920615943 bytes (  877.97 MB)
https://s3-api.dval.in/event/de-de/2022:  1348 files,   2879455536 bytes ( 2746.06 MB)
https://s3-api.dval.in/event/de-de/2023:  1285 files,   1716812169 bytes ( 1637.28 MB)
https://s3-api.dval.in/event/de-de/2024:   184 files,    224965098 bytes (  214.54 MB)
https://s3-api.dval.in/event/en-us/2020:   425 files,    896071072 bytes (  854.56 MB)
https://s3-api.dval.in/event/en-us/2021:  1346 files,   2642204517 bytes ( 2519.80 MB)
https://s3-api.dval.in/event/en-us/2022:  1368 files,   2856237402 bytes ( 2723.92 MB)
https://s3-api.dval.in/event/en-us/2023:  1296 files,   1681879611 bytes ( 1603.97 MB)
https://s3-api.dval.in/event/en-us/2024:   182 files,    221685968 bytes (  211.42 MB)
https://s3-api.dval.in/event/es-es/2021:   516 files,    917431278 bytes (  874.93 MB)
https://s3-api.dval.in/event/es-es/2022:  1346 files,   2845502273 bytes ( 2713.68 MB)
https://s3-api.dval.in/event/es-es/2023:  1261 files,   1720543780 bytes ( 1640.84 MB)
https://s3-api.dval.in/event/es-es/2024:   184 files,    223309668 bytes (  212.96 MB)
https://s3-api.dval.in/event/fr-fr/2021:   517 files,    929630286 bytes (  886.56 MB)
https://s3-api.dval.in/event/fr-fr/2022:  1329 files,   2805842297 bytes ( 2675.86 MB)
https://s3-api.dval.in/event/fr-fr/2023:  1269 files,   1705244962 bytes ( 1626.25 MB)
https://s3-api.dval.in/event/fr-fr/2024:   183 files,    223488393 bytes (  213.14 MB)
https://s3-api.dval.in/event/id-id/2021:   515 files,    931187277 bytes (  888.05 MB)
https://s3-api.dval.in/event/id-id/2022:  1339 files,   2816369156 bytes ( 2685.90 MB)
https://s3-api.dval.in/event/id-id/2023:  1262 files,   1716826847 bytes ( 1637.29 MB)
https://s3-api.dval.in/event/id-id/2024:   183 files,    225872213 bytes (  215.41 MB)
https://s3-api.dval.in/event/it-it/2022:   155 files,    322034036 bytes (  307.12 MB)
https://s3-api.dval.in/event/it-it/2023:  1268 files,   1709599154 bytes ( 1630.40 MB)
https://s3-api.dval.in/event/it-it/2024:   181 files,    224166590 bytes (  213.78 MB)
https://s3-api.dval.in/event/ja-jp/2022:  1068 files,   2311665079 bytes ( 2204.58 MB)
https://s3-api.dval.in/event/ja-jp/2023:  1345 files,   1851727143 bytes ( 1765.94 MB)
https://s3-api.dval.in/event/ja-jp/2024:   175 files,    222935506 bytes (  212.61 MB)
https://s3-api.dval.in/event/ko-kr/2021:   510 files,    876124965 bytes (  835.54 MB)
https://s3-api.dval.in/event/ko-kr/2022:  1349 files,   2776851634 bytes ( 2648.21 MB)
https://s3-api.dval.in/event/ko-kr/2023:  1250 files,   1750763120 bytes ( 1669.66 MB)
https://s3-api.dval.in/event/ko-kr/2024:   183 files,    226046103 bytes (  215.57 MB)
https://s3-api.dval.in/event/pt-pt/2021:   518 files,    912780193 bytes (  870.50 MB)
https://s3-api.dval.in/event/pt-pt/2022:  1348 files,   2851463705 bytes ( 2719.37 MB)
https://s3-api.dval.in/event/pt-pt/2023:  1272 files,   1714194703 bytes ( 1634.78 MB)
https://s3-api.dval.in/event/pt-pt/2024:   184 files,    224921032 bytes (  214.50 MB)
https://s3-api.dval.in/event/ru-ru/2021:   518 files,    915749773 bytes (  873.33 MB)
https://s3-api.dval.in/event/ru-ru/2022:  1330 files,   2806051298 bytes ( 2676.06 MB)
https://s3-api.dval.in/event/ru-ru/2023:  1254 files,   1702915925 bytes ( 1624.03 MB)
https://s3-api.dval.in/event/ru-ru/2024:   184 files,    224295282 bytes (  213.90 MB)
https://s3-api.dval.in/event/th-th/2021:   519 files,    921403501 bytes (  878.72 MB)
https://s3-api.dval.in/event/th-th/2022:  1332 files,   2757839289 bytes ( 2630.08 MB)
https://s3-api.dval.in/event/th-th/2023:  1258 files,   1741787799 bytes ( 1661.10 MB)
https://s3-api.dval.in/event/th-th/2024:   184 files,    224918641 bytes (  214.50 MB)
https://s3-api.dval.in/event/tr-tr/2022:   155 files,    328073565 bytes (  312.88 MB)
https://s3-api.dval.in/event/tr-tr/2023:  1265 files,   1726490494 bytes ( 1646.51 MB)
https://s3-api.dval.in/event/tr-tr/2024:   181 files,    218428151 bytes (  208.31 MB)
https://s3-api.dval.in/event/vi-vn/2021:   516 files,    920564845 bytes (  877.92 MB)
https://s3-api.dval.in/event/vi-vn/2022:  1343 files,   2832472162 bytes ( 2701.26 MB)
https://s3-api.dval.in/event/vi-vn/2023:  1252 files,   1701435336 bytes ( 1622.62 MB)
https://s3-api.dval.in/event/vi-vn/2024:   184 files,    224669102 bytes (  214.26 MB)
https://s3-api.dval.in/event/zh-cn/2021:   512 files,    926416028 bytes (  883.50 MB)
https://s3-api.dval.in/event/zh-cn/2022:  1334 files,   2678184114 bytes ( 2554.12 MB)
https://s3-api.dval.in/event/zh-cn/2023:  1260 files,   1713615355 bytes ( 1634.23 MB)
https://s3-api.dval.in/event/zh-cn/2024:   183 files,    228217727 bytes (  217.65 MB)
https://s3-api.dval.in/event/zh-tw/2021:   522 files,    937714876 bytes (  894.27 MB)
https://s3-api.dval.in/event/zh-tw/2022:  1343 files,   2808912378 bytes ( 2678.79 MB)
https://s3-api.dval.in/event/zh-tw/2023:  1258 files,   1688859693 bytes ( 1610.62 MB)
https://s3-api.dval.in/event/zh-tw/2024:   184 files,    228183383 bytes (  217.61 MB)
======================================================================================
Total      https://s3-api.dval.in/event: 46733 files,  79533647426 bytes (75849.20 MB)
                                                                         ( = 74.07 GB)

Signed-off-by: Samuel Giffard <[email protected]>
Signed-off-by: Samuel Giffard <[email protected]>
Signed-off-by: Samuel Giffard <[email protected]>
This comes with uploaded data.

Signed-off-by: Samuel Giffard <[email protected]>
@Mulugruntz Mulugruntz self-assigned this Feb 10, 2024
@Mulugruntz Mulugruntz linked an issue Feb 10, 2024 that may be closed by this pull request
@Mulugruntz Mulugruntz changed the base branch from main to sgiffard/35-events-manually-fix-malformed-urls February 10, 2024 14:47
@Mulugruntz Mulugruntz force-pushed the sgiffard/34-events-upload-all-images-on-minio-s3 branch from bbef888 to 666aef0 Compare February 15, 2024 21:23
@maalni maalni self-requested a review February 15, 2024 22:05
Base automatically changed from sgiffard/35-events-manually-fix-malformed-urls to main February 16, 2024 08:54
@Mulugruntz Mulugruntz merged commit d0c1324 into main Feb 16, 2024
@Mulugruntz Mulugruntz deleted the sgiffard/34-events-upload-all-images-on-minio-s3 branch February 16, 2024 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Events: Upload all images on MinIO S3
2 participants