Revise caching strategy

[johnbelamaric]

This contains the changes from kptdev/kpt#4094.

It needs more e2e tests though before merging.

[nephio-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnbelamaric

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnbelamaric]

/hold

This will pass the Porch E2E tests, but seems to still have some issues that need fixing. I need to create some new E2E tests that will fail due to those issues, and debug this before we merge it.

[efiacor]

/hold

This will pass the Porch E2E tests, but seems to still have some issues that need fixing. I need to create some new E2E tests that will fail due to those issues, and debug this before we merge it.

Hi @johnbelamaric
Could you maybe expand on what the "issues" are and maybe we could take a look at testing/addressing them?
As an aside, is the overall "plan" described in kptdev/kpt#3630 a major piece of work and how high a priority is it?

[efiacor]

/retest

[efiacor]

/test all

[johnbelamaric]

Umm. I don't remember the details from back in December. Our TNA team identified some issues in their testing so we backed this change out. @tliron can you reach out to the team and they can perhaps provide some insight?

[liamfallon]

This PR introduces a separate mutex for reconciling the cache, which does not block reads. Therefore reads can proceed even if there is a reconcile ongoing as long as the reconcile is not actually updating the cache or the repo. It also consolidates the cache recnciliation into a new method, which does a 3-way reconciliation against the cache, what's in the CRDs in etc and what's in Git.

The implementation in this PR was merged manually into the current main and the performance of Porch with this PR compared with the current main is shown on the plot below. The change introduced in #129 increases the performance of Porch much more so than this PR does.

Therefore I suggest we close this PR.

#135 is a manual rebase of this PR against the current main branch.