refactor(metadb): change backend serialization to protobuf #1842

laurentiuNiculae · 2023-09-21T17:04:14Z

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

andaaron

My partial review.

Makefile

errors/errors.go

pkg/api/controller.go

pkg/common/oci.go

pkg/meta/types/types.go

pkg/meta/boltdb/proto_boltdb.go

pkg/meta/types/types.go

pkg/extensions/search/convert/metadb.go

codecov · 2023-10-13T13:03:26Z

Codecov Report

Merging #1842 (bf8e5a5) into main (f34af3c) will decrease coverage by 2.06%.
The diff coverage is 85.52%.

@@            Coverage Diff             @@
##             main    #1842      +/-   ##
==========================================
- Coverage   91.93%   89.88%   -2.06%     
==========================================
  Files         155      158       +3     
  Lines       26807    26978     +171     
==========================================
- Hits        24646    24249     -397     
- Misses       1600     2009     +409     
- Partials      561      720     +159

Files	Coverage Δ
errors/errors.go	`100.00% <ø> (ø)`
pkg/api/routes.go	`94.50% <100.00%> (ø)`
pkg/common/common.go	`95.38% <100.00%> (+0.14%)`	⬆️
pkg/extensions/imagetrust/image_trust.go	`90.84% <100.00%> (-0.24%)`	⬇️
pkg/extensions/imagetrust/image_trust_disabled.go	`100.00% <ø> (ø)`
pkg/extensions/search/convert/oci.go	`94.00% <ø> (-0.83%)`	⬇️
pkg/extensions/search/cve/scan.go	`100.00% <100.00%> (ø)`
pkg/extensions/sync/references/cosign.go	`97.03% <100.00%> (+2.55%)`	⬆️
pkg/extensions/sync/references/oci.go	`91.66% <100.00%> (+1.84%)`	⬆️
pkg/extensions/sync/references/referrers_tag.go	`89.52% <100.00%> (-3.58%)`	⬇️
... and 22 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pkg/api/controller_test.go

pkg/extensions/imagetrust/image_trust.go

pkg/extensions/imagetrust/image_trust_disabled.go

pkg/test/oci-utils/repo.go

pkg/test/mocks/repo_db_mock.go

pkg/extensions/search/resolver.go

pkg/extensions/search/convert/metadb.go

pkg/meta/dynamodb/dynamodb.go

pkg/meta/proto/meta.proto

Makefile

.github/workflows/golangci-lint.yaml

Makefile

pkg/meta/proto/config.proto

pkg/meta/proto_go/config.pb.go

pkg/meta/proto/config.proto

pkg/test/deprecated/deprecated.go

.github/workflows/golangci-lint.yaml

.gitignore

Makefile

codecov.yml

Makefile

pkg/extensions/search/convert/convert_internal_test.go

errors/errors.go

Use protocol buffers and update the metadb interface to better suit our search needs Signed-off-by: Ramkumar Chinchani <[email protected]> Signed-off-by: Laurentiu Niculae <[email protected]>

rchincha

lgtm

laurentiuNiculae changed the title ~~Protobuff metadb~~ [WIP]: Protobuff metadb Sep 21, 2023

laurentiuNiculae force-pushed the protobuff-metadb branch 4 times, most recently from 52b5b32 to 8761abb Compare October 6, 2023 14:08

laurentiuNiculae marked this pull request as ready for review October 6, 2023 14:08

laurentiuNiculae requested review from shimish2, rchincha, andaaron and eusebiu-constantin-petu-dbk as code owners October 6, 2023 14:08

laurentiuNiculae force-pushed the protobuff-metadb branch 2 times, most recently from 6e604de to 97b37b8 Compare October 6, 2023 15:57

andaaron reviewed Oct 12, 2023

View reviewed changes

Makefile Show resolved Hide resolved

errors/errors.go Outdated Show resolved Hide resolved

pkg/api/controller.go Outdated Show resolved Hide resolved

pkg/common/oci.go Outdated Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 6 times, most recently from 414ce16 to 058f79a Compare October 13, 2023 10:52

andaaron reviewed Oct 13, 2023

View reviewed changes

pkg/extensions/search/convert/metadb.go Outdated Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 2 times, most recently from cb7fe4d to e199e83 Compare October 13, 2023 12:21

andaaron reviewed Oct 13, 2023

View reviewed changes

laurentiuNiculae force-pushed the protobuff-metadb branch from c8d4e2f to 8839281 Compare October 16, 2023 18:59

rchincha reviewed Oct 16, 2023

View reviewed changes

pkg/meta/proto/meta.proto Outdated Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 3 times, most recently from 3cf330b to 7f28bee Compare October 17, 2023 12:02

laurentiuNiculae force-pushed the protobuff-metadb branch 6 times, most recently from 50dccdf to 209da66 Compare October 24, 2023 15:23

adodon2go reviewed Oct 25, 2023

View reviewed changes

Makefile Show resolved Hide resolved

Makefile Show resolved Hide resolved

.github/workflows/golangci-lint.yaml Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 3 times, most recently from 5e819cc to ddd5627 Compare October 25, 2023 14:41

adodon2go assigned laurentiuNiculae Oct 25, 2023

adodon2go linked an issue Oct 25, 2023 that may be closed by this pull request

[Refactor][Investigate]: MetaDB: Use protobuf for data encoding instead of json #1831

Closed

laurentiuNiculae force-pushed the protobuff-metadb branch from ddd5627 to 37d5ea1 Compare October 25, 2023 15:39

rchincha reviewed Oct 25, 2023

View reviewed changes

pkg/meta/proto/config.proto Outdated Show resolved Hide resolved

pkg/meta/proto_go/config.pb.go Outdated Show resolved Hide resolved

pkg/meta/proto/config.proto Outdated Show resolved Hide resolved

pkg/test/deprecated/deprecated.go Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 4 times, most recently from e3ca46b to 771a996 Compare October 26, 2023 15:15

adodon2go requested changes Oct 27, 2023

View reviewed changes

.github/workflows/golangci-lint.yaml Outdated Show resolved Hide resolved

.gitignore Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

codecov.yml Outdated Show resolved Hide resolved

adodon2go reviewed Oct 27, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

adodon2go requested changes Oct 27, 2023

View reviewed changes

pkg/extensions/search/convert/convert_internal_test.go Show resolved Hide resolved

laurentiuNiculae force-pushed the protobuff-metadb branch 3 times, most recently from b78db62 to 5801e14 Compare October 30, 2023 10:11

adodon2go reviewed Oct 30, 2023

View reviewed changes

errors/errors.go Outdated Show resolved Hide resolved

refactor(metadb): improve UX by speeding up metadb serialize/deserialize

bf8e5a5

Use protocol buffers and update the metadb interface to better suit our search needs Signed-off-by: Ramkumar Chinchani <[email protected]> Signed-off-by: Laurentiu Niculae <[email protected]>

laurentiuNiculae force-pushed the protobuff-metadb branch from 5801e14 to bf8e5a5 Compare October 30, 2023 14:27

rchincha approved these changes Oct 30, 2023

View reviewed changes

rchincha merged commit 56ad9e6 into project-zot:main Oct 30, 2023
32 of 34 checks passed

rchincha mentioned this pull request Nov 1, 2023

feat: improve UX by speeding up metadb serialize/deserialize #1829

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(metadb): change backend serialization to protobuf #1842

refactor(metadb): change backend serialization to protobuf #1842

laurentiuNiculae commented Sep 21, 2023

andaaron left a comment

codecov bot commented Oct 13, 2023 •

edited

Loading

rchincha left a comment

refactor(metadb): change backend serialization to protobuf #1842

refactor(metadb): change backend serialization to protobuf #1842

Conversation

laurentiuNiculae commented Sep 21, 2023

andaaron left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 13, 2023 • edited Loading

Codecov Report

rchincha left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 13, 2023 •

edited

Loading