feat: Add NewListDirectoryPathsPager for azuredatalake

[paulbrittain]

azure-sdk-for-go issue: #23852
other relevant issue: Azure/azure-rest-api-specs#31894

@tanyasethi-msft Please review 🙂

[github-actions]

Thank you for your contribution @paulbrittain! We will review the pull request and get back to you soon.

[tanyasethi-msft]

/azp run go - azdatalake

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tanyasethi-msft]

I also ran the live test (unrecorded, which runs live on a storage account) pipeline, and it seems to fail for the new test added. Find the error below -
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=4455746&view=logs&j=30081788-26d0-562e-2814-e633030d4b74&t=d5c76e6b-9ce1-5a1d-ad9d-4395c3c4169b&l=50981

[paulbrittain]

I also ran the live test (unrecorded, which runs live on a storage account) pipeline, and it seems to fail for the new test added. Find the error below - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=4455746&view=logs&j=30081788-26d0-562e-2814-e633030d4b74&t=d5c76e6b-9ce1-5a1d-ad9d-4395c3c4169b&l=50981

I've added more tests to client_test.go.

I took a closer look at the failure: The tests are adding a Test directory inside the Storage container, so when a test adds a dir1 directory, the actual directory list without a prefix is:

Test
Test/dir1

When a prefix of Test is given, as in the newly added TestFilesystemListDeletedPathsWithPrefix and in other ...WithPrefix tests, the directory list should be:

dir1

The expected results have been changed to expect this.

Could you please run the azdatalake pipeline again?

[tanyasethi-msft]

/azp run go - azdatalake

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tanyasethi-msft]

@paulbrittain ,
Here are the results -
--- FAIL: Test/TestFilesystemListDirectoryPathsContinuation (9.09s)
(Server Failure) - I shall look into this and get back here.
--- PASS: Test/TestFilesystemListDirectoryPathsMaxResults (0.18s)
--- FAIL: Test/TestFilesystemListDirectoryPathsWithPrefix (0.19s)

[tanyasethi-msft]

/azp run go - azdatalake

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tanyasethi-msft]

I tried to fix the pipeline by making changes to the changelog file, but I do not have access to push to your fork.

This change in changelog file and removal of live test TestFilesystemListDirectoryPathsContinuation will fix the pipeline.

Error in the test from backend -
Line 1858 -> pager := fsClient.NewListDirectoryPathsPager(&opts) [Normal Listing - showonly=directory]
RequestUrl='https://xxx.blob.core.windows.net:443/gofsfilesystemlistdirectorypathscontinuation?comp=list&maxResults=2&restype=container&showonly=directories'
Line 1866 -> pager = fsClient.NewListDeletedPathsPager() [Listing with showonly=deleted with CT from previous Listing, hence fails]

[paulbrittain]

Hi @tanyasethi-msft

Thank you very much for your continued support, and my apologies for being slow in updating the branch. The suggested changes have been implemented now.

Please check the pipeline again when you have the chance

Paul

[paulbrittain]

@microsoft-github-policy-service agree company="Helio AG"

[tanyasethi-msft]

/azp run go - azdatalake

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tanyasethi-msft]

Hey @paulbrittain ,
As I mentioned earlier, this test - TestFilesystemListDirectoryPathsContinuation is not checking the required information and can be removed. This is also causing the pipeline failure.
Please do the needful and then let's merge the changes to main. We have a release scheduled today - if this can go in today, it'd be great.
Otherwise, we'd have to wait a month for this to GA