Skip to content

Commit

Permalink
Sync release/v1.7 to main (#2495)
Browse files Browse the repository at this point in the history
* fix: git add chart directory for release (#2356)

Signed-off-by: hlts2 <[email protected]>

* 🔖 🤖 Release v1.7.11 (#2360) (#2361)

* 🔖 🤖 Release v1.7.11



* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 578746e according to the output
from Gofumpt and Prettier.

Details: #2360

---------

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Change docker scan timeout longer (#2363)

* refactor code using golangci-lint (#2362)

Signed-off-by: kpango <[email protected]>

* Create SECURITY.md (#2367)

* Create SECURITY.md

Signed-off-by: Yusuke Kato <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e8609b0 according to the output
from Gofumpt and Prettier.

Details: #2367

---------

Signed-off-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* add commit hash build image (#2359)

Signed-off-by: kpango <[email protected]>

* Make agent export index metrics to Pod k8s resource (#2319)

* Add config

* Add clusterRole and serviceAccount for agent

* Add more metrics

* Format

* Add export index info duration configuration

* Refactor internal/k8s

* Move apply function to internal/k8s

* Add more config

* Remove unused import and variable

* Use vald errors

* Add pod name and pod namespace to hack e2e agent

* Apply tagalign

* Revert unnecesarry specification of PodName

* Enable export index info to Kubernetes in agent templates only when the config is enabled

* Fix agent clusterrole name

* Update key name

* Update comments

* Add index count as export metrics

* Ignore gomnd

* Add index count entry to exportMetricsOnCreateIndex function

* Update values schema

* Fix formatter job

* Remove unused defines

* update docker build target platform selection rules (#2370)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Update workflow to release readreplica chart (#2383)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Add readreplica chart release

* update k8s/readreplica

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 70f1dad according to the output
from Gofumpt and Prettier.

Details: #2383

* [bugfix] fix backport github value bypass failure (#2378)

Signed-off-by: kpango <[email protected]>

* Revert backport.yaml

---------

Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* 💚 ♻️ Add Con-Bench helm chart to the Vald charts (#2388)

Signed-off-by: vankichi <[email protected]>

* Delete unnecessary code for mirror (#2366)

* fix: delete unnecessary code for mirror

Signed-off-by: hlts2 <[email protected]>

* fix: make deps/install and format

Signed-off-by: hlts2 <[email protected]>

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* fix: naming of variable

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* change JP logo to EN logo (#2369)

Signed-off-by: kpango <[email protected]>
Co-authored-by: taisuou <[email protected]>

* Add rotate-all option to rotator (#2305)

* Add rotate-all option to rotator

* Fix linter warnings

* Make it possible to choose multiple rotation target

* Add tests for parseReplicaID

* Add tests for parseReplicaID

* Hide testify in internal

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in dd84674 according to the output
from Gofumpt and Prettier.

Details: #2305

* Apply format

* Lint

* Lint

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>

* Resolve kvs already closed before last saving (#2390)

Signed-off-by: kpango <[email protected]>

* create continous benchmark doc (#2352)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* 📝 create continous benchmark doc

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in ace001a according to the output
from Gofumpt and Prettier.

Details: #2352

* ♻️ rafactor default params

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 9880dfc according to the output
from Gofumpt and Prettier.

Details: #2352

* ♻️ Fix format

Signed-off-by: vankichi <[email protected]>

* Revert "improve Backport Pull Request Title & Description Quality (#2373)"

This reverts commit 4246def.

* ♻️ remove dimension from con-bench

Signed-off-by: vankichi <[email protected]>

* 📝 Add sample benchmark manifest

Signed-off-by: vankichi <[email protected]>

* Update docs/performance/continuos-benchmatk.md

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

* Update docs/performance/continuos-benchmatk.md

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

---------

Signed-off-by: kpango <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* fix: build error of internal kvs test (#2396)

Signed-off-by: hlts2 <[email protected]>

* 🤖 Update license headers / Format Go codes and YAML files (#2397)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* fix: disable protobuf dispatch for client (#2401)

Signed-off-by: hlts2 <[email protected]>

* update deps (#2404)

Signed-off-by: kpango <[email protected]>

* 🔖 🤖 Release v1.7.12 (#2408) (#2409)

* 🔖 🤖 Release v1.7.12



* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 5bd8605 according to the output
from Gofumpt and Prettier.

Details: #2408

---------

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Add tests for index information export (#2412)

* Add tests for index information export

* Add comments

* 📝 Fix typo of file name (#2413)

Signed-off-by: vankichi <[email protected]>

* Fix agent-faiss build failed (#2418)

* 🐳 Fix agent-faiss build failed

Signed-off-by: vankichi <[email protected]>

* 🐳 Fix COPY binary

Signed-off-by: vankichi <[email protected]>

* 🐳 Change COPY binary (use ld-linux-x86-64)

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* Fix the logic to determine docker image (#2410)

* fix: update action to get tag list

Signed-off-by: hlts2 <[email protected]>

* fix: refactor sub command execution

Signed-off-by: hlts2 <[email protected]>

* fix: use tag list in detect ci container workflow

Signed-off-by: hlts2 <[email protected]>

* fix: name for detected result

Signed-off-by: hlts2 <[email protected]>

* fix: invalid variable bug

Signed-off-by: hlts2 <[email protected]>

* fix: execution error and invalid variable

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary comment

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary parameters

Signed-off-by: hlts2 <[email protected]>

* fix: use new determine tag logic in the docker build workflow

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix extra tags handling and small refactor

Signed-off-by: hlts2 <[email protected]>

* fix: build tag rule or release branch based on tag list

Signed-off-by: hlts2 <[email protected]>

* fix: typo

Signed-off-by: hlts2 <[email protected]>

* fix: indent

Signed-off-by: hlts2 <[email protected]>

* fix: refactor syntax

Signed-off-by: hlts2 <[email protected]>

* fix: typo and invalid option error

Signed-off-by: hlts2 <[email protected]>

* fix: use helm extra option

Signed-off-by: hlts2 <[email protected]>

* fix: add comment for tag management

Signed-off-by: hlts2 <[email protected]>

* fix: unsetbug of helm extra option

Signed-off-by: hlts2 <[email protected]>

* fix: build target condition

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Update build rule for nightly image (#2421)

* fix: update build rule for nightly

Signed-off-by: hlts2 <[email protected]>

* fix: indent and deleted unnecessary tag

Signed-off-by: hlts2 <[email protected]>

* fix: update tag name when tag is unknown

Signed-off-by: hlts2 <[email protected]>

* fix: add secondary tags for build performance

Signed-off-by: hlts2 <[email protected]>

* fix: argument order

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Add `index-operator` template implementation (#2375)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* [bugfix] fix backport github value bypass failure (#2378)

Signed-off-by: kpango <[email protected]>

* [bugfix] fix backport github value bypass failure (#2381)

Signed-off-by: kpango <[email protected]>

* Add index operator boilerplate

* Add k8s controller

* Add pod and job controllers

* Add annotations in the pod reconciler

* Update auto indexing service configurations

* Update WithOnReconcileFunc signature to include context

* Refactor debugging log

* Refactor reconcile funcs

* add nolint

* remove unused parameters

* lint

* Add dockerfile and build ci

* remove params

* Add index-operator image build command to Makefile

* Add index operator k8s templates

* Update helm schema

* Update readme

* Add internal/k8s as a trigger for images that depends on controller-runtime

* remove comment

* Update pkg/index/operator/service/operator.go

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Update pkg/index/operator/service/operator.go

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Refactor

* Add build commands

* Refactor

Co-authored-by: Yusuke Kato <[email protected]>

---------

Signed-off-by: kpango <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Fix output settings to determine-docker-image-tag action and release branch build tag name (#2423)

* fix: add outut settings for action

Signed-off-by: hlts2 <[email protected]>

* fix: tag name for release branch build

Signed-off-by: hlts2 <[email protected]>

* fix: refactor

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* fix: typo of execution rule (#2426)

Signed-off-by: hlts2 <[email protected]>

* Add flush API for deleting all vectors (#2273)

* add flush api

Signed-off-by: aknishid <[email protected]>

* added flush api

* fix typo

* fix pb.go & fix conflict

Signed-off-by: aknishid <[email protected]>

* add flush api

* add handler Flush

Signed-off-by: aknishid <[email protected]>

* add vald.Client flush

* fix filter gateway

Signed-off-by: aknishid <[email protected]>

* fix handler cnts

Signed-off-by: aknishid <[email protected]>

* delete unused var

Signed-off-by: aknishid <[email protected]>

* add author

Signed-off-by: Yamada Takuya <[email protected]>

* fix error

Signed-off-by: aknishid <[email protected]>

* fix flush.go & ngt.go

Signed-off-by: aknishid <[email protected]>

* add flush to example

Signed-off-by: aknishid <[email protected]>

* fix gateway lb

Signed-off-by: aknishid <[email protected]>

* update flush proto

Signed-off-by: aknishid <[email protected]>

* fix docs api README.md

Signed-off-by: aknishid <[email protected]>

* Fix flush ctx

Signed-off-by: aknishid <[email protected]>

* update docs & comment

Signed-off-by: aknishid <[email protected]>

* after make format

Signed-off-by: aknishid <[email protected]>

* add e2e

Signed-off-by: aknishid <[email protected]>

* update example & tutorial

Signed-off-by: aknishid <[email protected]>

* Update internal/errors/ngt.go

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* fix review

Signed-off-by: aknishid <[email protected]>

* modify status.md

Signed-off-by: aknishid <[email protected]>

* add error handling

Signed-off-by: aknishid <[email protected]>

* update proto

Signed-off-by: aknishid <[email protected]>

* fix doc

Signed-off-by: aknishid <[email protected]>

* ♻️ Fix proto

Signed-off-by: vankichi <[email protected]>

* use atomic bool

Signed-off-by: aknishid <[email protected]>

* fix proto

Signed-off-by: aknishid <[email protected]>

* update proto

Signed-off-by: aknishid <[email protected]>

* fix gateway error

Signed-off-by: aknishid <[email protected]>

* ✅ 🚧 Avoid e2e failing due to unimplemented handler and Add unimplmented handler method for the future

Signed-off-by: vankichi <[email protected]>

* Update apis/proto/v1/payload/payload.proto

Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: aknishid <[email protected]>

---------

Signed-off-by: aknishid <[email protected]>
Signed-off-by: Yamada Takuya <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: aknishid <[email protected]>
Co-authored-by: takuyaymd <[email protected]>
Co-authored-by: takuyyam <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* docs: add aknishid as a contributor for code, and maintenance (#2431)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in abc4c52 according to the output
from Gofumpt and Prettier.

Details: #2431

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* update deps & add validation for Flush API when agent is Read Only (#2433)

Signed-off-by: kpango <[email protected]>

* Update README.md (grammer) (#2440)

Signed-off-by: Hrichik Mazumder <[email protected]>

* docs: add hrichiksite as a contributor for doc (#2441)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in b8054d9 according to the output
from Gofumpt and Prettier.

Details: #2441

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Added snapshot timestamp annotations to read replica agent (#2428)

* Add lastTimeRotateReplicaTimestampAnnotationsKey constant

* Use ptr package

* Update annotations in ngt.go and rotator.go

* Add labels to Pod struct and update annotations handling***

***Introduce vald package for k8s related constants***

***Implement rotateIfNeeded function to check if read replica rotation is needed

* Refactor

* set sample log level debug

* Use k8s timestamp

* Refactor

* Format

* Add MatchingLabels method to ValdK8sClientMock

* Format

* Fix test

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 39b66fa according to the output
from Gofumpt and Prettier.

Details: #2428

* Rename constants

Co-authored-by: Yusuke Kato <[email protected]>

* Refactor rotateIfNeeded function to use label selector for listing deployments

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 6ace761 according to the output
from Gofumpt and Prettier.

Details: #2428

* FIx deepsource

* Revert

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>

* fix: bugfix version update for docker build (#2445)

Signed-off-by: hlts2 <[email protected]>

* Fix index job logic to pass DNS A record (#2438)

* fix: choose user given target address in priority

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary code

Signed-off-by: hlts2 <[email protected]>

* fix: add logic to connect the user given target addresses

Signed-off-by: hlts2 <[email protected]>

* fix: add info log about new connection target

Signed-off-by: hlts2 <[email protected]>

* fix: indent of configuration data

Signed-off-by: hlts2 <[email protected]>

* fix: indent bug

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Fix operator-sdk version (#2447)

* fix: down grade operator version

Signed-off-by: hlts2 <[email protected]>

* fix: upgrade operator-sdk version

Signed-off-by: hlts2 <[email protected]>

* fix: operator-sdk version

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* add file name lint (#2417)

Signed-off-by: kpango <[email protected]>

* fix: add extra option for ci-container build (#2451)

Signed-off-by: hlts2 <[email protected]>

* Add base of benchmark operator dashboard (#2430)

* 📈 Add base of benchmark operator dashboard

Signed-off-by: vankichi <[email protected]>

* 💚 Fix build benchmark images trigger

Signed-off-by: vankichi <[email protected]>

* 💚 Add info tag for info metrics

Signed-off-by: vankichi <[email protected]>

* 📈 showing job image and tags

Signed-off-by: vankichi <[email protected]>

* 📈 add bench job status

Signed-off-by: vankichi <[email protected]>

* ♻️ Update dashboard and rename

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

* Update internal/observability/metrics/tools/benchmark/benchmark.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Implement index operator logic for read replica rotation (#2444)

* Remove job reconciler

* Add leader election to index operator

* Add roles for leader election

* Add operator logic for rotation

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 844cca1 according to the output
from Gofumpt and Prettier.

Details: #2444

* Remove unnecesary change

* Fix build

* nits

* Refactor

* Fix clusterrole

* Move to podv2

* Add job concurrency check

* Refactor

* Refactor

* Fix heavy copying

* Update helm schema

* Fix package doc

* Remove unused code and refactor For() method

* Update target read replica ID in config

* Update internal/k8s/podv2/option.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Refactor

* rename podv2 to v2/pod

* Add client DI

* Fix envkey helper

* Set namespace from values.yaml

* Use annotations to specify the id

* Update schema

* Update sample

* Update values

* Update schema

* Fix spelling

* Fix package name

* Fix read replica e2e

* Add job templates for rotation, creation, saving, and correction

* Update config.go to decode k8s object

* Update sample.yaml for index operator

* Use job template from cfg

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 584d1f0 according to the output
from Gofumpt and Prettier.

Details: #2444

* Fix tagalign

* Fix config test

* Fix config test

* Disable read replica e2e for now

---------

Signed-off-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* add inner product distance type for ngt (#2454)

Signed-off-by: kpango <[email protected]>

* Fix e2e for read replica and add e2e for index operator (#2455)

* Fix read replica e2e

* Update default image list in action.yaml

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 6cdac9f according to the output
from Gofumpt and Prettier.

Details: #2455

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Add unit tests for index operator (#2460)

* Add prototype tests

* Refactor

* Fix not to use kubeconfig

* 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2457)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Bugfix recreate benchmark job when operator reboot (#2463)

* 🐛 Fix recreate benchmark job already runnnig when operator rebooted

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix benchmarkJobReconcile status handling

Signed-off-by: vankichi <[email protected]>

* ✅ Fix test

Signed-off-by: vankichi <[email protected]>

* ♻️ Update k8s dir

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* Refactor k8s types (#2462)

* Refactor types

* Refactor hdf5_test.go

* Format

* Refactor

* add extra flags for NGT build phase (#2467)

* Fix workflow trigger for backport pr creation (#2471)

* Automatically add backport main label for release-pr (#2473)

* feat: automatically add backport main label for release-pr

Signed-off-by: Hiroto Funakoshi <[email protected]>

* fix: deleted unnecessary quotation

Signed-off-by: Hiroto Funakoshi <[email protected]>

---------

Signed-off-by: Hiroto Funakoshi <[email protected]>

* update deps (#2468)

* update deps

Signed-off-by: kpango <[email protected]>

* fix server_test.go due to the grpc-go updates

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: kpango <[email protected]>

* Update README.md (#2479)

Signed-off-by: Yusuke Kato <[email protected]>

* Implement client metrics interceptor for continuous benchmark job (#2477)

* ♻️ Mount benchmark-operator-config to vald-benchmark-job Pod

Signed-off-by: vankichi <[email protected]>

* 📈 add framework for send gRPC client metrics

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in a532b7d according to the output
from Gofumpt and Prettier.

Details: #2477

* 📈 Add client metric interceptor

Signed-off-by: vankichi <[email protected]>

* ♻️ Change vbor chart for apply general job configuration

Signed-off-by: vankichi <[email protected]>

* Update pkg/tools/benchmark/job/usecase/benchmarkd.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

* 🐛 Fix build error and add comment

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix format

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* implement ngt-rs and its unit test and add GitHub Actions workflows (#2478)

* implement test

* update rust dependencies

* fix

* make format

* add Github Actions workflows

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: Yusuke Kato <[email protected]>

* 📈 Add client metrics panels for continuous benchmark job (#2481)

Signed-off-by: vankichi <[email protected]>

* Add rust versions (#2482)

* add rust versions

* fix

* add rust version

* add rust-toolchain.toml

* make format

* Update continuous benchmark docs (#2485)

* 📝 Update continous benchmark docs

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e3ab1d2 according to the output
from Gofumpt and Prettier.

Details: #2485

* Update docs/performance/continuous-benchmark.md

Signed-off-by: Kiichiro YUKAWA <[email protected]>

* 📝 Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* update NGT version (#2484)

* fix sed line (#2488)

* 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2487)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add efficient search logic for large top-k users (#2491)

* add efficient search logic for large top-k users

Signed-off-by: kpango <[email protected]>

* fix bug

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Signed-off-by: Vdaas CI <[email protected]>
Signed-off-by: kpango <[email protected]>
Signed-off-by: Yusuke Kato <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>
Signed-off-by: aknishid <[email protected]>
Signed-off-by: Yamada Takuya <[email protected]>
Signed-off-by: aknishid <[email protected]>
Signed-off-by: Hrichik Mazumder <[email protected]>
Signed-off-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: vdaas-ci <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: taisho <[email protected]>
Co-authored-by: taisuou <[email protected]>
Co-authored-by: aknishid <[email protected]>
Co-authored-by: takuyaymd <[email protected]>
Co-authored-by: takuyyam <[email protected]>
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Hrichik Mazumder <[email protected]>
  • Loading branch information
14 people authored May 8, 2024
1 parent 9efb3aa commit a698f3f
Show file tree
Hide file tree
Showing 118 changed files with 4,646 additions and 1,696 deletions.
14 changes: 14 additions & 0 deletions .gitfiles
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@
.github/workflows/coverage.yml
.github/workflows/detect-internal-config-changes.yml
.github/workflows/dockers-agent-faiss-image.yml
.github/workflows/dockers-agent-image.yml
.github/workflows/dockers-agent-ngt-image.yml
.github/workflows/dockers-agent-sidecar-image.yml
.github/workflows/dockers-benchmark-job-image.yml
Expand Down Expand Up @@ -167,8 +168,10 @@ apis/grpc/v1/mirror/mirror.pb.go
apis/grpc/v1/mirror/mirror_vtproto.pb.go
apis/grpc/v1/payload/interface.go
apis/grpc/v1/payload/payload.pb.go
apis/grpc/v1/payload/payload.pb.json.go
apis/grpc/v1/payload/payload_vtproto.pb.go
apis/grpc/v1/rpc/errdetails/error_details.pb.go
apis/grpc/v1/rpc/errdetails/error_details.pb.json.go
apis/grpc/v1/rpc/errdetails/error_details_vtproto.pb.go
apis/grpc/v1/vald/filter.pb.go
apis/grpc/v1/vald/filter_vtproto.pb.go
Expand Down Expand Up @@ -243,6 +246,7 @@ assets/docs/overview/search_flow.svg
assets/docs/overview/update_flow.svg
assets/docs/overview/upsert_flow.svg
assets/docs/overview/vald_basic_architecture.svg
assets/docs/performance/benchmark-grafana.png
assets/docs/troubleshooting/provisioning_flow_chart.drawio
assets/docs/troubleshooting/provisioning_flow_chart.svg
assets/docs/tutorial/getstarted.svg
Expand Down Expand Up @@ -505,6 +509,8 @@ design/Vald Architecture Assets.drawio
design/Vald Architecture Dataflow.drawio
design/Vald Architecture Overview.drawio
design/vald_architecture_overview.png
dockers/agent/core/agent/Dockerfile
dockers/agent/core/agent/README.md
dockers/agent/core/faiss/Dockerfile
dockers/agent/core/ngt/Dockerfile
dockers/agent/core/ngt/README.md
Expand Down Expand Up @@ -1121,6 +1127,7 @@ internal/net/grpc/handler.go
internal/net/grpc/health/health.go
internal/net/grpc/health/health_test.go
internal/net/grpc/interceptor.go
internal/net/grpc/interceptor/client/metric/metric.go
internal/net/grpc/interceptor/client/trace/trace.go
internal/net/grpc/interceptor/server/logging/accesslog.go
internal/net/grpc/interceptor/server/logging/accesslog_test.go
Expand Down Expand Up @@ -1814,6 +1821,11 @@ rust/Cargo.lock
rust/Cargo.toml
rust/bin/agent/Cargo.toml
rust/bin/agent/src/main.rs
rust/libs/ngt-rs/Cargo.toml
rust/libs/ngt-rs/build.rs
rust/libs/ngt-rs/src/input.cpp
rust/libs/ngt-rs/src/input.h
rust/libs/ngt-rs/src/lib.rs
rust/libs/ngt/Cargo.toml
rust/libs/ngt/src/lib.rs
rust/libs/proto/Cargo.toml
Expand All @@ -1836,6 +1848,7 @@ rust/libs/proto/src/sidecar.v1.rs
rust/libs/proto/src/sidecar.v1.tonic.rs
rust/libs/proto/src/vald.v1.rs
rust/libs/proto/src/vald.v1.tonic.rs
rust/rust-toolchain.toml
tests/chaos/chart/.helmignore
tests/chaos/chart/Chart.yaml
tests/chaos/chart/README.md
Expand Down Expand Up @@ -1881,6 +1894,7 @@ versions/OTEL_OPERATOR_VERSION
versions/PROMETHEUS_STACK_VERSION
versions/PROTOBUF_VERSION
versions/REVIEWDOG_VERSION
versions/RUST_VERSION
versions/TELEPRESENCE_VERSION
versions/VALDCLI_VERSION
versions/VALD_VERSION
Expand Down
7 changes: 4 additions & 3 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ assignees: ""

<!--- Please change the versions below along with your environment -->

- Go Version: 1.22.2
- Go Version: 1.22.3
- Rust Version: 1.77.2
- Docker Version: 20.10.8
- Kubernetes Version: v1.29.3
- NGT Version: 2.2
- Kubernetes Version: v1.30.0
- NGT Version: 2.2.1
7 changes: 4 additions & 3 deletions .github/ISSUE_TEMPLATE/security_issue_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ assignees: ""

<!--- Please change the versions below along with your environment -->

- Go Version: 1.22.2
- Go Version: 1.22.3
- Rust Version: 1.77.2
- Docker Version: 20.10.8
- Kubernetes Version: v1.29.3
- NGT Version: 2.2
- Kubernetes Version: v1.30.0
- NGT Version: 2.2.1
7 changes: 4 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,11 @@

<!--- Please change the versions below along with your environment -->

- Go Version: 1.22.2
- Go Version: 1.22.3
- Rust Version: 1.77.2
- Docker Version: 20.10.8
- Kubernetes Version: v1.29.3
- NGT Version: 2.2
- Kubernetes Version: v1.30.0
- NGT Version: 2.2.1

### Checklist

Expand Down
74 changes: 74 additions & 0 deletions .github/workflows/dockers-agent-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
name: "Build docker image: agent"
on:
push:
branches:
- "main"
- "release/v*.*"
- "!release/v*.*.*"
tags:
- "*.*.*"
- "v*.*.*"
- "*.*.*-*"
- "v*.*.*-*"
paths:
- ".github/actions/docker-build/action.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-agent-image.yml"
- "rust/Cargo.lock"
- "rust/Cargo.toml"
- "rust/bin/agent/**"
- "rust/libs/ngt/**"
- "rust/libs/ngt-rs/**"
- "rust/libs/proto/**"
- "dockers/agent/core/ngt-rust/Dockerfile"
- "versions/RUST_VERSION"
- "versions/NGT_VERSION"
pull_request:
paths:
- ".github/actions/docker-build/action.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-agent-image.yml"
- "rust/Cargo.lock"
- "rust/Cargo.toml"
- "rust/bin/agent/**"
- "rust/libs/ngt/**"
- "rust/libs/ngt-rs/**"
- "rust/libs/proto/**"
- "dockers/agent/core/agent/Dockerfile"
- "versions/RUST_VERSION"
- "versions/NGT_VERSION"
pull_request_target:
paths:
- ".github/actions/docker-build/action.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-agent-image.yml"
- "rust/Cargo.lock"
- "rust/Cargo.toml"
- "rust/bin/agent/**"
- "rust/libs/ngt/**"
- "rust/libs/ngt-rs/**"
- "rust/libs/proto/**"
- "dockers/agent/core/agent/Dockerfile"
- "versions/RUST_VERSION"
- "versions/NGT_VERSION"
jobs:
build:
uses: ./.github/workflows/_docker-image.yaml
with:
target: agent
secrets: inherit
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ on:
- "versions/GO_VERSION"
- "versions/NGT_VERSION"
- "versions/KUBECTL_VERSION"
- "versions/RUST_VERSION"
jobs:
dump-contexts-to-log:
runs-on: ubuntu-latest
Expand Down
22 changes: 17 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ GHCRORG = ghcr.io/$(ORG)/$(NAME)
AGENT_NGT_IMAGE = $(NAME)-agent-ngt
AGENT_FAISS_IMAGE = $(NAME)-agent-faiss
AGENT_SIDECAR_IMAGE = $(NAME)-agent-sidecar
AGENT_IMAGE = $(NAME)-agent
CI_CONTAINER_IMAGE = $(NAME)-ci-container
DEV_CONTAINER_IMAGE = $(NAME)-dev-container
DISCOVERER_IMAGE = $(NAME)-discoverer-k8s
Expand Down Expand Up @@ -58,9 +59,12 @@ GO_CLEAN_DEPS := true
GOTEST_TIMEOUT = 30m
CGO_ENABLED = 1

RUST_HOME = /usr/local/lib/rust
RUSTUP_HOME = $(RUST_HOME)/rustup
CARGO_HOME = $(RUST_HOME)/cargo
RUST_HOME ?= /usr/local/lib/rust
RUSTUP_HOME ?= $(RUST_HOME)/rustup
CARGO_HOME ?= $(RUST_HOME)/cargo
RUST_VERSION := $(eval RUST_VERSION := $(shell cat versions/RUST_VERSION))$(RUST_VERSION)

NPM_GLOBAL_PREFIX := $(eval NPM_GLOBAL_PREFIX := $(shell npm prefix --location=global))$(NPM_GLOBAL_PREFIX)

NPM_GLOBAL_PREFIX := $(eval NPM_GLOBAL_PREFIX := $(shell npm prefix --location=global))$(NPM_GLOBAL_PREFIX)

Expand Down Expand Up @@ -138,14 +142,17 @@ ifeq ($(GOARCH),amd64)
CFLAGS ?= -mno-avx512f -mno-avx512dq -mno-avx512cd -mno-avx512bw -mno-avx512vl
CXXFLAGS ?= $(CFLAGS)
EXTLDFLAGS ?= -m64
NGT_EXTRA_FLAGS ?=
else ifeq ($(GOARCH),arm64)
CFLAGS ?=
CXXFLAGS ?= $(CFLAGS)
EXTLDFLAGS ?= -march=armv8-a
NGT_EXTRA_FLAGS ?=
else
CFLAGS ?=
CXXFLAGS ?= $(CFLAGS)
EXTLDFLAGS ?=
NGT_EXTRA_FLAGS ?=
endif

BENCH_DATASET_MD5S := $(eval BENCH_DATASET_MD5S := $(shell find $(BENCH_DATASET_MD5_DIR) -type f -regex ".*\.md5"))$(BENCH_DATASET_MD5S)
Expand Down Expand Up @@ -535,6 +542,11 @@ version/vald:
version/go:
@echo $(GO_VERSION)

.PHONY: version/rust
## print rust version
version/rust:
@echo $(RUST_VERSION)

.PHONY: version/ngt
## print NGT version
version/ngt:
Expand Down Expand Up @@ -571,7 +583,7 @@ ngt/install: /usr/local/include/NGT/Capi.h
/usr/local/include/NGT/Capi.h:
git clone --depth 1 --branch v$(NGT_VERSION) https://github.com/yahoojapan/NGT $(TEMP_DIR)/NGT-$(NGT_VERSION)
cd $(TEMP_DIR)/NGT-$(NGT_VERSION) && \
cmake -DCMAKE_C_FLAGS="$(CFLAGS)" -DCMAKE_CXX_FLAGS="$(CXXFLAGS)" .
cmake -DCMAKE_C_FLAGS="$(CFLAGS)" -DCMAKE_CXX_FLAGS="$(CXXFLAGS)" "$(NGT_EXTRA_FLAGS)" .
make -j -C $(TEMP_DIR)/NGT-$(NGT_VERSION)
make install -C $(TEMP_DIR)/NGT-$(NGT_VERSION)
cd $(ROOTDIR)
Expand All @@ -582,7 +594,7 @@ ngt/install: /usr/local/include/NGT/Capi.h
## install Faiss
faiss/install: /usr/local/lib/libfaiss.so
/usr/local/lib/libfaiss.so:
curl -LO https://github.com/facebookresearch/faiss/archive/v$(FAISS_VERSION).tar.gz
curl -fsSLO https://github.com/facebookresearch/faiss/archive/v$(FAISS_VERSION).tar.gz
tar zxf v$(FAISS_VERSION).tar.gz -C $(TEMP_DIR)/
cd $(TEMP_DIR)/faiss-$(FAISS_VERSION) && \
cmake -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -B build . && \
Expand Down
6 changes: 6 additions & 0 deletions Makefile.d/build.mk
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,12 @@ cmd/tools/benchmark/operator/operator:
$(eval CGO_ENABLED = 0)
$(call go-build,tools/benchmark/operator,,-static,,,$@)

rust/target/release/agent:
pushd rust && cargo build -p agent --release && popd

rust/target/debug/agent:
pushd rust && cargo build -p agent && popd

.PHONY: binary/build/zip
## build all binaries and zip them
binary/build/zip: \
Expand Down
4 changes: 2 additions & 2 deletions Makefile.d/client.mk
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@ valdcli/install: $(BINDIR)/valdcli
ifeq ($(UNAME),Darwin)
$(BINDIR)/valdcli:
mkdir -p $(BINDIR)
curl -LO https://github.com/rinx/vald-client-clj/releases/download/$(VALDCLI_VERSION)/valdcli-macos.zip
curl -fsSLO https://github.com/rinx/vald-client-clj/releases/download/$(VALDCLI_VERSION)/valdcli-macos.zip
unzip valdcli-macos.zip
rm -f valdcli-macos.zip
mv valdcli $(BINDIR)/valdcli
else
$(BINDIR)/valdcli:
mkdir -p $(BINDIR)
curl -LO https://github.com/rinx/vald-client-clj/releases/download/$(VALDCLI_VERSION)/valdcli-linux-static.zip
curl -fsSLO https://github.com/rinx/vald-client-clj/releases/download/$(VALDCLI_VERSION)/valdcli-linux-static.zip
unzip valdcli-linux-static.zip
rm -f valdcli-linux-static.zip
mv valdcli $(BINDIR)/valdcli
Expand Down
Loading

0 comments on commit a698f3f

Please sign in to comment.