New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Xload prefetch initial code #1585

Merged

souravgupta-msft merged 63 commits into feature/xload from sourav/prefetch

Feb 3, 2025

+1,542 −17

Member

souravgupta-msft commented Dec 4, 2024 •

edited by jainakanksha-msft

Loading

✅ What

Initial code for prefetch feature in Xload component

🤔 Why

Download all the blobs on mount

NOTE

Please note that this the first PR for prefetch. Below items are in TODO list and will be added in future PRs

Stats collection
Open, close, getAttr and read dir calls in xload component
UT and E2E
Retry mechanism

Related Links

https://microsoft.sharepoint.com/:w:/t/AzureStorage/EYpVETudnXJNiYSiYDwRiR0BycWuL10UOK3g32eiHubEWQ?e=TqEFWP

vibhansa-msft and others added 30 commits

April 12, 2024 15:45


          Creating component

7ed70c3


          Cleanup

88ec516


          cleanup'

3cbfc73


          Updating copyright statement

5019e96


          Adding block

1dc590e


          Adding block and blockpool

0f14e2a


          Merge branch 'feature/xload' of https://github.com/Azure/azure-storag…

1c7e914

…e-fuse into feature/xload


          Adding threadpool

4a54134


          Base code

c309894


          Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…

c60bad0

…to feature/xload


          changing mode type to enum

d0bfd1b


          changing mode type to enum (#1386)

cad5756


          Adding splitter logic

141a549


          local listing

741a0ca


          Added splitter logic

2329c4b


          Correct error handling

dc31b1e


          Sync with lister

3d9357f


          Updated

0e90768


          Error handling

84b562e


          Adding remote lister

60cb00e


          log correction

68e4bd8


          sample

a7b545f


          lister

3faa5f2


          adding xlister

360bbec


          Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…

e153b3f

…to sourav/xload


          Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…

73a444c

…to sourav/xload


          Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…

a5eca7a

…to sourav/xload


          check

1f43c23


          Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…

a840414

…to sourav/xload


          temp

93f0224

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/lister.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/lister.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/data_manager.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/xload.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

cmd/imports.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/xload.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/xload.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Outdated Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/common/block.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/data_manager.go Outdated Show resolved Hide resolved


          Adding fio config to generate files in parallel

e1db656

vibhansa-msft modified the milestones: v2-2.4.1, v2-2.5.0


          review comments

fde22ea

souravgupta-msft commented

View reviewed changes

component/xload/comp/splitter.go Outdated Show resolved Hide resolved

souravgupta-msft added 2 commits

January 28, 2025 12:29


          review comments

a0f7ad8


          review comments

f792e7d

vibhansa-msft approved these changes

View reviewed changes

souravgupta-msft added 3 commits

January 30, 2025 15:38


          review comments

7a9010b


          update mode and lmt as per blob properties

4b34833


          change

cce3a95

vibhansa-msft reviewed

View reviewed changes

component/xload/comp/splitter.go Outdated Show resolved Hide resolved


          file mode

1e139aa

souravgupta-msft merged commit 9e16cb4 into feature/xload

5 checks passed

souravgupta-msft deleted the sourav/prefetch branch

February 3, 2025 06:38

jainakanksha-msft reviewed

View reviewed changes

component/xload/common/block.go Show resolved Hide resolved

component/azstorage/azstorage.go Show resolved Hide resolved

component/azstorage/azstorage.go Show resolved Hide resolved

component/xload/common/utils.go Show resolved Hide resolved

component/xload/common/xcomponent.go Show resolved Hide resolved

component/xload/comp/lister.go Show resolved Hide resolved

component/xload/comp/splitter.go Show resolved Hide resolved

component/xload/xload.go Show resolved Hide resolved

component/xload/xload.go Show resolved Hide resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

jainakanksha-msft jainakanksha-msft left review comments

dphulkar-msft dphulkar-msft left review comments

vibhansa-msft vibhansa-msft approved these changes

ashruti-msft Awaiting requested review from ashruti-msft ashruti-msft is a code owner

syeleti-msft Awaiting requested review from syeleti-msft syeleti-msft is a code owner

Labels

pr-ready-for-review V2 xload