[PRETEST 1, 2, 3]: Added an overall proposal to implement cache support using bare repo for managing third-party dependencies

[Manoramsharma]

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

• N
• Y

ref #384

2. What is the scope of this PR (e.g. component or file name):

proposal/add_support_for_cache.md

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Other

Here I have proposed an overall implementation of kpm managing the third-party dependencies making the use of bare repository, serving as a cache and facilitating the downloading of package dependencies from the bare repo itself. The implementation talks about handling the file systems based on the bare repo directory . But it is not that much solid for now I need reviews form the kcl maintainers to move forward with the exact changes to achieve the discussed approach.

cc: @Peefy @zong-zhe

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

• N
• Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

• Unit test
• Integration test
• Benchmark (add benchmark stats below)
• Manual test (add detailed scripts or steps below)
• Other

[coveralls]

Pull Request Test Coverage Report for Build 10496585466

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 40.551%

---

Totals
Change from base Build 10482352953:	0.0%
Covered Lines:	3182
Relevant Lines:	7847

---

💛 - Coveralls

[Manoramsharma]

Hi @zong-zhe I have developed a final documentation for enhancing the KPM's dependencies management. This is with reference to the rust package manager Cargo, the way it handles the dependencies management through bare repo. I have also submitted a research work for different languages and the unified system supported by them.

I need review and valuable feedbacks from the maintainers and members of kcl community on the design doc so that I can quickly start working on the pretests based on the discussed approach.

Thanks

cc: @Peefy @liangyuanpeng

[zong-zhe]

Here you can try to provide more details, you can refer to the visitor

The kcl package visitor - https://github.com/kcl-lang/kpm/blob/main/pkg/client/visitor.go
The visitor usage -

kpm/pkg/client/run.go

Line 528 in 2b1e0b9

err = NewVisitor(*pkgSource, c).Visit(pkgSource, func(kclPkg *pkg.KclPkg) error {

visitor does not provide caching package, it will immediately download and load the package. You can extend this visitor to support the file storage structure designed earlier with a caching mechanism.

[Manoramsharma]

By here you mean for extending our file system to achieve a unified dependency management, right? As required in Pretest-3.

[zong-zhe]

Yes, you can extend this Visitor to complete the final file system,

Now the visitor is work as:

KCL package source -> Visitor (download the remote package and load it) -> KCL Package

You can extend Visitor to use the local file storage structure designed earlier:

KCL package source -> Visitor (download, cache and load with new local file storage structure) -> KCL Package

[zong-zhe]

Hi @Manoramsharma 😄

I just merged a PR and left a TODO for you, you can try to complete this TODO:

kpm/pkg/downloader/downloader.go

Line 124 in 655022e

// TODO: After the new local storage structure is complete,

I extended DownloadOption with two new fields that support whether caching is enabled and the cache path.

kpm/pkg/downloader/downloader.go

Line 26 in 655022e

CachePath string

kpm/pkg/downloader/downloader.go

Line 29 in 655022e

EnableCache bool

The work you need to do next now consists of two main parts:

1. For GitDownloader, if EnableCache is True, you will use CachePath as the path to the bare repository, and then clone the bare repository from the CachePath to the LocalPath directory.

2. For OciDownloader, if EnableCache is True, you will use CachePath as the path to OCI KCL package tar, and untar the file from CachePath to LocalPath.

[zong-zhe]

Do not delete the previous content in this part, you can rewrite this part of the content to align with the PR, but do not delete.