From fbd13eedca2af5152b159673389e613f685478b3 Mon Sep 17 00:00:00 2001 From: Calvin Neo Date: Mon, 30 May 2022 16:59:30 +0800 Subject: [PATCH] Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy (#72) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ctl: validate column family name when scanning data using ctl (#11269) * ctl: should check column name validation when scan data using ctl close #11268 Signed-off-by: yuqi1129 * fix format problem Signed-off-by: yuqi1129 Co-authored-by: Xinye Tao * txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977) * Add mutation checking logic Signed-off-by: MyonKeminta * Fix build Signed-off-by: MyonKeminta * Fix usages of mutation Signed-off-by: MyonKeminta * Update kvproto and renaming Signed-off-by: MyonKeminta * Add a simple test Signed-off-by: MyonKeminta * Fix incorrect assertion Signed-off-by: MyonKeminta * Support check existance during pessimistic lock Signed-off-by: MyonKeminta * Add assertion level support Signed-off-by: MyonKeminta * Add tests Signed-off-by: MyonKeminta * Add more test cases Signed-off-by: MyonKeminta * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta * Fix build Signed-off-by: MyonKeminta * ref #11542 Signed-off-by: MyonKeminta * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta Co-authored-by: MyonKeminta Co-authored-by: Ti Chi Robot * metrics: fix memory metrics (#11656) Signed-off-by: Shuaipeng Yu Co-authored-by: Ti Chi Robot * coprocesser: refactor encoding and add gbk_chinese_ci map (#11661) * close #11660 Signed-off-by: xiongjiwei * close #11660 Signed-off-by: xiongjiwei Co-authored-by: Ti Chi Robot * storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu * wip Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu * fix enable_ttl Signed-off-by: andylokandy * simplify test Signed-off-by: andylokandy * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu Co-authored-by: andylokandy Co-authored-by: Ti Chi Robot * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng * a more comment Signed-off-by: qupeng * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy * refactor if let to match Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy Co-authored-by: Ti Chi Robot * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <786537003@qq.com> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 Co-authored-by: Ti Chi Robot * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 * fix lint Signed-off-by: MuZhou233 * fix clippy Signed-off-by: MuZhou233 * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 * fix lint Signed-off-by: MuZhou233 * Integrate thread_io Signed-off-by: MuZhou233 * record bytes in request Signed-off-by: MuZhou233 * fix logical bug Signed-off-by: MuZhou233 * Update test write_bytes Signed-off-by: MuZhou233 * Delete unused & Rename func Signed-off-by: MuZhou233 * Add test for read & Update test for write Signed-off-by: MuZhou233 * Add buffered thread io Signed-off-by: MuZhou233 * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 * Change fetch logic. close #10867 Signed-off-by: MuZhou233 * Add AtomicIOBytes Signed-off-by: MuZhou233 * Split ThreadIOSentinel Signed-off-by: MuZhou233 * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 * Adjust code style & fix Signed-off-by: MuZhou233 * improve write_bytes test Signed-off-by: MuZhou233 Co-authored-by: Ti Chi Robot * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi * RAII removes on_reporter_closing Signed-off-by: Zhenchi * fmt Signed-off-by: Zhenchi * fix build ref #11691 Signed-off-by: Zhenchi * datasink -> data_sink Signed-off-by: Zhenchi * fmt Signed-off-by: Zhenchi * fix build Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy * fix test Signed-off-by: andylokandy * add test Signed-off-by: andylokandy * fix clippy Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy * fmt Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy * address comment Signed-off-by: andylokandy * rename vars Signed-off-by: andylokandy * improve vars Signed-off-by: andylokandy Co-authored-by: Ti Chi Robot * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen * Fix bugs Signed-off-by: Peng Guanwen * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen * remove redundant log Signed-off-by: Peng Guanwen * Show the entire key Signed-off-by: Peng Guanwen * Fix lint issue Signed-off-by: Peng Guanwen * resolve conflict Signed-off-by: andylokandy Co-authored-by: Andy Lok * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen * br/stream: make clippy happy Signed-off-by: yujuncen * br/stream: fix tests Signed-off-by: Yu Juncen * backup: fix a dummy bug Signed-off-by: Yu Juncen * backup: add config of io-threads Signed-off-by: yujuncen * backup: ref #11350: some minior change Signed-off-by: yujuncen * backup: added some metrics Signed-off-by: yujuncen * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen * backup: added some slow log and metrics Signed-off-by: yujuncen * backup: address comments Signed-off-by: yujuncen * backup: fix build Signed-off-by: yujuncen * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen * backup: address comments Signed-off-by: yujuncen * backup: always set IO type to Export Signed-off-by: yujuncen * backup:make clippy happy Signed-off-by: yujuncen * backup: address comments Signed-off-by: yujuncen * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen * Reformat code Signed-off-by: Peng Guanwen * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen Co-authored-by: Andy Lok * format code Signed-off-by: Peng Guanwen Co-authored-by: Andy Lok * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 Co-authored-by: Yilin Chen * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei * close #11735, and address comments Signed-off-by: xufei * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen * ref #11000: change the default backup threads. Signed-off-by: yujuncen * backup: run tasks in background Signed-off-by: yujuncen * backup: run rustfmt Signed-off-by: yujuncen * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen * br-stream: fix build Signed-off-by: yujuncen * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer * s3: add grafana json Signed-off-by: 3pointer * br: add config for s3 multi part upload Signed-off-by: 3pointer * update comment Signed-off-by: 3pointer * update comment Signed-off-by: 3pointer * address comment Signed-off-by: 3pointer * hidden the new config Signed-off-by: 3pointer * only hidden config in config-template.toml Signed-off-by: 3pointer * address comment Signed-off-by: 3pointer * close #11727 && format Signed-off-by: 3pointer * fix Signed-off-by: 3pointer Co-authored-by: Ti Chi Robot * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi * wrap config notifier Signed-off-by: Zhenchi * add unit tests Signed-off-by: Zhenchi * polish ref #11691 Signed-off-by: Zhenchi * address comments Signed-off-by: Zhenchi * fix test Signed-off-by: Zhenchi * fix outdated comments Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie * address comments Signed-off-by: tabokie * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx * implement unit test for global config in pd client Signed-off-by: lemonhx * resolved formatting issues Signed-off-by: lemonhx * impl pd client asyncly Signed-off-by: lemonhx * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx * resove issues Signed-off-by: lemonhx * clippy Signed-off-by: lemonhx * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx Co-authored-by: Zhenchi * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 * format code Signed-off-by: you06 * add test Signed-off-by: you06 * address comment Signed-off-by: you06 * add an integration test Signed-off-by: you06 * format code Signed-off-by: you06 Co-authored-by: Ti Chi Robot * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge * fix case. close #11495 Signed-off-by: guo-shaoge * fix comment Signed-off-by: guo-shaoge * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi * wrap config notifier Signed-off-by: Zhenchi * add unit tests Signed-off-by: Zhenchi * polish ref #11691 Signed-off-by: Zhenchi * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi * fmt & try to resolve #11689 Signed-off-by: Zhenchi * add unit tests ref #11691 Signed-off-by: Zhenchi * retrigger test Signed-off-by: Zhenchi * address comments Signed-off-by: Zhenchi * fix test Signed-off-by: Zhenchi * fix outdated comments Signed-off-by: Zhenchi * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi Co-authored-by: Yexiang Zhang * address comment Signed-off-by: Zhenchi * remove enabled Signed-off-by: Zhenchi * fix Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot Co-authored-by: Wenxuan Co-authored-by: Yexiang Zhang * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx * Fix test compilation Signed-off-by: mornyx * Fix test compilation Signed-off-by: mornyx * Fix test compilation Signed-off-by: mornyx * Use master kvproto Signed-off-by: mornyx Co-authored-by: Ti Chi Robot * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi * fix build Signed-off-by: Zhenchi * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi Co-authored-by: Wenxuan * fix build Signed-off-by: Zhenchi Co-authored-by: Wenxuan * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx * Modify comments Signed-off-by: mornyx * Fix pd.rs Signed-off-by: mornyx * Refactor Signed-off-by: mornyx * Remove import Signed-off-by: mornyx * max_resource_groups 200 -> 100 Signed-off-by: mornyx * Fix ut Signed-off-by: mornyx * Use iterator to avoid clone Signed-off-by: mornyx * Optimize close #11754 Signed-off-by: mornyx Co-authored-by: Ti Chi Robot * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee * remove raftstore wait Signed-off-by: Jay Lee * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee * remove useless field Signed-off-by: Jay Lee * address comment Signed-off-by: Jay Lee * fix unstable raw get case Signed-off-by: Jay Lee Co-authored-by: Liqi Geng Co-authored-by: Ti Chi Robot * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth * modify the priority of the azure login methods Signed-off-by: Leavrth * fix feature match branch Signed-off-by: Leavrth * implement token update retriable Signed-off-by: Leavrth * fix some problems Signed-off-by: Leavrth * close #11731 Signed-off-by: Leavrth * rustfmt Signed-off-by: Leavrth * make clippy success Signed-off-by: Leavrth * simplify uploader Signed-off-by: Leavrth * commit some suggestions Signed-off-by: Leavrth * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei Co-authored-by: Ti Chi Robot * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei Co-authored-by: Ti Chi Robot * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie * manual clippy Signed-off-by: tabokie * format and clippy Signed-off-by: tabokie * enable default features when running clippy, ref #11119 Signed-off-by: tabokie * fix build Signed-off-by: tabokie * revert box_collection lint Signed-off-by: tabokie * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao Co-authored-by: Lei Zhao Co-authored-by: Lei Zhao * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen * remove potentially incorrect assert Signed-off-by: Yilin Chen * check term and version when reading the lock table Signed-off-by: Yilin Chen * revert a log change Signed-off-by: Yilin Chen * change term when becoming follower Signed-off-by: Yilin Chen * fix lint Signed-off-by: Yilin Chen * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen * fix another test Signed-off-by: Yilin Chen * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen * fix a tiny comment error Signed-off-by: Yilin Chen * make clippy happy Signed-off-by: Yilin Chen * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx Co-authored-by: Ti Chi Robot * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang Co-authored-by: Xinye Tao Co-authored-by: Ti Chi Robot * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee * address comment Signed-off-by: Jay Lee * declare platform compatability Signed-off-by: Jay Lee Co-authored-by: Ti Chi Robot * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 * resolve format problem in discussion Signed-off-by: yuqi1129 * resolve format problem in discussion again Signed-off-by: yuqi1129 Co-authored-by: Xinye Tao Co-authored-by: Ti Chi Robot * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers * check lease before sending read index Signed-off-by: 5kbpers * renew lease on raft tick to cover lease till next tick Signed-off-by: linning * refactor Signed-off-by: linning * don't use inspect_lease Signed-off-by: linning * fix ci Signed-off-by: linning * add test Signed-off-by: linning * remove unneed comment Signed-off-by: linning * close #5388 Signed-off-by: 5kbpers * check pending_read_count before hibernating Signed-off-by: 5kbpers * increase the range to check lease expired Signed-off-by: 5kbpers * renew leader lease after not hibernate Signed-off-by: 5kbpers * fix hibernate states metrics Signed-off-by: 5kbpers * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers * fix test Signed-off-by: 5kbpers * use a individual tick to renewing lease Signed-off-by: 5kbpers * fix condition Signed-off-by: 5kbpers * stablize test_node_renew_lease Signed-off-by: 5kbpers * cleanup unused changes Signed-off-by: 5kbpers * add config item for interval of tick Signed-off-by: 5kbpers * rename tick Signed-off-by: 5kbpers * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * count renew lease read index into raft metrics Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * disable the tick for test_renew_lease Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * fix lint error Signed-off-by: 5kbpers * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * make clippy happy Signed-off-by: 5kbpers Co-authored-by: linning Co-authored-by: Ti Chi Robot * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth * close#11812 Signed-off-by: Leavrth * use to_string Signed-off-by: Leavrth Co-authored-by: Ti Chi Robot * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie * fix error message Signed-off-by: tabokie * parse cpu quota in floats Signed-off-by: tabokie * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee * remove extra comment Signed-off-by: Jay Lee Co-authored-by: Ti Chi Robot * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers * add slow log Signed-off-by: 5kbpers * remove duplicate metric Signed-off-by: 5kbpers * add resolved ts metrics for leader peers Signed-off-by: 5kbpers * close #11767 Signed-off-by: 5kbpers * init endpoint with store meta Signed-off-by: 5kbpers Co-authored-by: Ti Chi Robot * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi * use atomic cell instead Signed-off-by: Zhenchi * debug assert Signed-off-by: Zhenchi Co-authored-by: Wenxuan * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 * update Signed-off-by: lhy1024 * ref #11521 Signed-off-by: lhy1024 * address comment Signed-off-by: lhy1024 * fix test Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou Co-authored-by: Xinye Tao Co-authored-by: Connor Co-authored-by: Ti Chi Robot * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi * address code review feedback -- close #11746 Signed-off-by: qi.xu * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu Co-authored-by: qi.xu * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee * fix format Signed-off-by: Jay Lee * fix clippy Signed-off-by: Jay Lee Co-authored-by: zhouqiang * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 * fix Signed-off-by: MuZhou233 * fix test Signed-off-by: MuZhou233 * fix test Signed-off-by: MuZhou233 Co-authored-by: Xinye Tao Co-authored-by: Ti Chi Robot * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen * use bool array for registry Signed-off-by: Yilin Chen * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen * add associated const for number of PeerTick types Signed-off-by: Yilin Chen * change naming Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen * fix merged upstream Signed-off-by: Yilin Chen * fix test Signed-off-by: Yilin Chen * fix typo Signed-off-by: Yilin Chen * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung * address the comment Signed-off-by: Ryan Leung * address the comment Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 * make format Signed-off-by: Connor1996 * address comment Signed-off-by: Connor1996 * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee * address comment Signed-off-by: Jay Lee * speed up destroy Signed-off-by: Jay Lee * fix compile Signed-off-by: Jay Lee * further speed up Signed-off-by: Jay Lee * revert test case configuration Signed-off-by: Jay Lee * address comment Signed-off-by: Jay Lee * address comment Signed-off-by: Jay Lee Co-authored-by: Ti Chi Robot * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu Co-authored-by: Mini256 Co-authored-by: Mini256 Co-authored-by: Xiaoguang Sun * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie * try fixing the underflow Signed-off-by: tabokie * mute underflow warning for raft metrics Signed-off-by: tabokie * clean up unused data members Signed-off-by: tabokie * format Signed-off-by: tabokie * step back to 2022-01-07 Signed-off-by: tabokie * readme: change images based on github theme (#11795) Signed-off-by: sloorush Co-authored-by: Ti Chi Robot * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning * add test case Signed-off-by: linning * make clippy happy Signed-off-by: linning * address comment Signed-off-by: linning Co-authored-by: Ti Chi Robot * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan Co-authored-by: Ti Chi Robot * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie * update one more commit Signed-off-by: tabokie * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen * remove check_is_busy totally Signed-off-by: Yilin Chen * do not use wildcard match in error handling Signed-off-by: Yilin Chen * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 * update kvproto Signed-off-by: Connor1996 * address comment Signed-off-by: Connor1996 * fix test build Signed-off-by: Connor1996 * fix clippy Signed-off-by: Connor1996 * limit capacity Signed-off-by: Connor1996 * update kvproto Signed-off-by: Connor1996 * call stop on worker Signed-off-by: Connor1996 * rename worker Signed-off-by: Connor1996 * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha * close #11917 Signed-off-by: Suhaha Co-authored-by: goroutine Co-authored-by: Connor * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning * add log Signed-off-by: linning * make clippy happy Signed-off-by: linning * ref #11579 Signed-off-by: 5kbpers * reset raft tick Signed-off-by: 5kbpers * set has ready Signed-off-by: 5kbpers * address comments Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * fix panic Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * address comment Signed-off-by: 5kbpers * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers Co-authored-by: 5kbpers * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx * Fix warns Signed-off-by: mornyx Co-authored-by: Ti Chi Robot * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu Co-authored-by: Xiaoguang Sun * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie * update raft engine Signed-off-by: tabokie * update raft-engine and fix tests Signed-off-by: tabokie * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta Co-authored-by: Ti Chi Robot * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List Tests - Manual test (add detailed scripts or steps below) `cargo check` ### Release note ```release-note None Signed-off-by: linning Co-authored-by: Ti Chi Robot * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List Tests - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note ```release-note None Signed-off-by: Mattias Jonsson Co-authored-by: Ti Chi Robot * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie Co-authored-by: Ti Chi Robot * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin Signed-off-by: qupeng Co-authored-by: hi-rustin * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer Co-authored-by: Ti Chi Robot * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot Co-authored-by: Jay * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot Co-authored-by: Yilin Chen * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer Co-authored-by: Ti Chi Robot * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer Co-authored-by: Ti Chi Robot * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie * address comment Signed-off-by: tabokie * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer Co-authored-by: Ti Chi Robot * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie Co-authored-by: Ti Chi Robot * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson Co-authored-by: Ti Chi Robot * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong Co-authored-by: Yilin Chen Co-authored-by: Ti Chi Robot * replication mode: sync state to pd (#11751) Signed-off-by: disksing Co-authored-by: Ti Chi Robot * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer Co-authored-by: Ti Chi Robot * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng Co-authored-by: Ti Chi Robot * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee Co-authored-by: Ti Chi Robot * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong Co-authored-by: Ti Chi Robot * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu Co-authored-by: qi.xu Co-authored-by: Ti Chi Robot * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu Co-authored-by: Andy Lok Co-authored-by: Ti Chi Robot * *: fix some typos (#12066) Signed-off-by: cuishuang Co-authored-by: Ti Chi Robot * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy Co-authored-by: Xinye Tao * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot * github: Remove the pingcap/tidb-ansible from the PR template (#12102) close tikv/tikv#12103 Signed-off-by: hi-rustin Co-authored-by: Ti Chi Robot * copr: extract reschedule to inject a fail point (#12108) Signed-off-by: Zhenchi * raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883) ref tikv/tikv#11452 Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * *: check memory locks for replica read only on the leader (#12115) close tikv/tikv#12109 Consider a reader sends read index to the leader it supposed to be, but when the leader receives the read index message, it has stepped down to a follower. Without this commit, a peer will check the locks and re-fill the read index context with the result no matter what role it is. So, when the read index request is redirected to the new leader again, it no longer carries the context of lock checking. This commmit changes to only do the check when the peer is a leader. Then, the read index request will remain unchanged before being redirected to the leader. If the lease of the leader expires, it is still safe to check on the uncertained leader. If the heartbeat check passes and no new leader is elected, then the check works fine. If there is a new leader, when the old leader becomes a follower, it will clear its pending read index. Then, the reader has to resend the read index again to the correct leader. So, generally it is safe as long as we guarantee the check only happens on the leader. Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * apiv2 rawkv: logical deletion flag (#12090) ref tikv/tikv#11965 Signed-off-by: haojinming Co-authored-by: Ti Chi Robot * copr: add projection (#10689) close tikv/tikv#12114 Signed-off-by: ichn-hu Co-authored-by: Ti Chi Robot Co-authored-by: Alex Chi * raftstore: adjust raft max msg size dynamically (#12018) close tikv/tikv#12017 Signed-off-by: glorv Co-authored-by: Ti Chi Robot * *: update deps (#12098) - Upgrade grpcio to 0.10 - Upgrade openssl to 1.1.1m - Remove dependency on prost Signed-off-by: Jay Lee * tests: make raftstore tests about transactions more stable (#12122) close tikv/tikv#12120 A new fail point "after_propose" is added to insert callback from the test. Then we can ensure we release other key fail points after they take effect. The other change is to retry request automatically to avoid stale command errors. Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * metrics: support multi k8s in grafana dashboards (#12008) close tikv/tikv#12104 Signed-off-by: just1900 * storage: set in-memory pessimistic lock feature gate to 6.0.0 (#12078) ref tikv/tikv#11452 The in-memory pessimistic lock feature should be only enabled after all TiKV instances have been upgraded to the new version. Otherwise, transferring leader may be blocked because of the new protocol and procedure when in-memory pessimistic locks exist. So, we need to use a feature gate to make sure this feature is only enabled after all TiKVs have been upgraded to the new version. Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * raftstore,cdc: pass leader transferee to cdc observer (#12124) Signed-off-by: qupeng Co-authored-by: Ti Chi Robot * config: simplify handling of nullable configurations (#12133) Support deriving `OnlineConfig` on any `Option`. One caveat is such `T` must implement `Default`. * simplify handling of nullable configurations Signed-off-by: tabokie * add credit for copied code Signed-off-by: tabokie * patch unit test Signed-off-by: tabokie Co-authored-by: Ti Chi Robot * raftstore: support dynamic adjust max_batch_size (#11974) close tikv/tikv#11982 Signed-off-by: glorv * server: disallow coexist of raft engine and raftdb (#12047) * disallow coexist of raft engine and raftdb Signed-off-by: tabokie * add defer block Signed-off-by: tabokie * minor cleanup Signed-off-by: tabokie * check not clear Signed-off-by: tabokie * print out rm failure Signed-off-by: tabokie * print log for non-empty target engine Signed-off-by: tabokie * even more logs Signed-off-by: tabokie * use a state machine to monitor the switch Signed-off-by: tabokie * fix and add test Signed-off-by: tabokie * clippy Signed-off-by: tabokie * address comments Signed-off-by: tabokie * address comments Signed-off-by: tabokie * remove tempfile from engine_traits Signed-off-by: tabokie * address comments Signed-off-by: tabokie Co-authored-by: Ti Chi Robot * *: add more metrics and logs for in-memory pessimistic locking (#12089) ref tikv/tikv#11452 It adds logs when proposing locks during transferring leader and moving locks to new split regions. The metrics about the memory used by pessimistic locks and how many locks fail to be inserted due to reaching the memory limit are displayed in the grafana. Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * *: notify snapshot apply (#12126) close tikv/tikv#12057 In the past, it relies on base tick to check if a snapshot is applied. If the base tick is set to a very large value, it will take a long time to detect snapshot is finished. This PR adds a casual message to notify peer actively. Signed-off-by: Jay Lee Co-authored-by: Ti Chi Robot * txn: Fix assertion fail on non-pessimistic key with conflict (#12123) ref tikv/tikv#12113 txn: Fix assertion fail on non-pessimistic key with conflict (ref #12113) Signed-off-by: MyonKeminta Co-authored-by: Ti Chi Robot * cmd: link c++ explicitly (#12145) close tikv/tikv#12144 In the past, we rely on librocksdb-sys to include the required stdc++ library. This is not reliable after upgrading grpcio, which also needs the same library. I suspect it doesn't include all the required symbols. Anyway, this PR chooses to link to stdc++ explicitly in binary. Signed-off-by: Jay Lee * Fix logic of error string match in `bad-ssts` (#12049) Signed-off-by: Xintao Co-authored-by: Ti Chi Robot * APIV2: Implement RawMvccSnapshot & RawMvccIterator (#12087) ref tikv/tikv#11965 Signed-off-by: haojinming * *: add tip in contributing.md (#12129) close tikv/tikv#12142 polish contributing.md Signed-off-by: jackwener Co-authored-by: Ti Chi Robot * dr-ausosync: report min resolved ts (#12052) ref tikv/pd#4686 Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot * txn: support read-consistency read with tso checking (#12097) Signed-off-by: cfzjywxk * raftstore: collect statistics of buckets (#12080) ref tikv/tikv#11718 Collect read/write flow of buckets. Signed-off-by: 5kbpers Co-authored-by: Ti Chi Robot * config: enable in-memory pessimistic locks by default (#12153) close tikv/tikv#11452 Signed-off-by: Yilin Chen Co-authored-by: Ti Chi Robot * rsmeter: add integration tests for cpu (#12140) Signed-off-by: Zhenchi Co-authored-by: Ti Chi Robot * cdc: don't emit resolved timestamps before scan (#12156) Signed-off-by: qupeng * resolved_ts: fix failpoints (#12170) Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot * cdc: add a metrics for count of pending tasks (#12169) Signed-off-by: qupeng * raft_log_engine: fix managedfs create encrypted file (#12163) close tikv/tikv#12162 Signed-off-by: glorv Co-authored-by: Xinye Tao * config: limit backup workers cap (#12141) ref #11804 Signed-off-by: Wenbo Zhang Co-authored-by: Ti Chi Robot * split_controller: refine the sample function (#12177) split_controller: refine the sample function Signed-off-by: JmPotato * cdc: remove useless deps and fix typos (#12165) close tikv/tikv#12183 Remove usless deps and fix typos Signed-off-by: hi-rustin Co-authored-by: Ti Chi Robot * storage: support scale store scheduler pool size (#12068) close tikv/tikv#12067 Signed-off-by: glorv * *: update jemalloc version to fix profiling problem (#12179) close tikv/tikv#12180 *: upgrade jemalloc version to fix sampling problem during profiling Signed-off-by: cosven Co-authored-by: Jay * split_controller: refine and add some test cases for sample function (#12189) ref tikv/tikv#12185 split_controller: refine and add some test cases for sample function Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * Foreground Quota Limiter: limit the cpu, bandwidth to provide stable QPS (#12151) close tikv/tikv#11855, ref tikv/tikv#11856 Add a global forefront quota limiter based speed limiter to record the CPU resource overhead occupied by various requests. When the limiter reaches the limit quota, the request will be forced to sleep for a period of time to compensate. Statistics are only done on the part of process and are therefore approximate. Signed-off-by: tonyxuqqi Signed-off-by: qi.xu Signed-off-by: zhangjinpeng1987 Signed-off-by: Xintao Co-authored-by: tonyxuqqi Co-authored-by: qi.xu Co-authored-by: zhangjinpeng1987 * UnifiedReadPool: support adjusting the pool size dynamically (#11783) close tikv/tikv#11781 Signed-off-by: Wenbo Zhang * raftstore: drop raft message by replication state (#12076) close tikv/tikv#12074 Signed-off-by: disksing * copr: return buckets version for unary call (#12184) Signed-off-by: youjiali1995 * tikv-ctl: support raft engine ctl (#12019) * tikv-ctl: support raft engine ctl Signed-off-by: tabokie * kebab case Signed-off-by: tabokie * fix build Signed-off-by: tabokie * switch to 0.1.x branch Signed-off-by: tabokie Co-authored-by: Ti Chi Robot * dr-ausosync: make report interval configurable (#12157) ref tikv/pd#4686 Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot * resolved_ts: fix oom and add more metrics (#12190) * add metrics Signed-off-by: 5kbpers * fix Signed-off-by: 5kbpers * remove unused code Signed-off-by: 5kbpers * add timeout Signed-off-by: 5kbpers Co-authored-by: qupeng Co-authored-by: Ti Chi Robot * avoid to banish region leaders when rolling upgrade (#12167) Signed-off-by: qupeng Co-authored-by: Ti Chi Robot * rocksdb: use new pipelined-write to replace multibatch-write (#12060) close tikv/tikv#12059 Signed-off-by: Little-Wallace * add vdso to the pprof blocklist (#12201) (#12209) close tikv/tikv#9765, ref tikv/tikv#12201 Signed-off-by: ti-srebot Co-authored-by: YangKeao * raftstore:Fix memory metrics overflow on raft msgs (#12186) (#12224) close tikv/tikv#12160, ref tikv/tikv#12186 Fix raft msg memory metrics overflow. Signed-off-by: ti-srebot Co-authored-by: tier-cap Co-authored-by: Ti Chi Robot * do not force compact unreplicated entries (#12181) (#12228) close tikv/tikv#12161, ref tikv/tikv#12181 Signed-off-by: ti-srebot Co-authored-by: Xinye Tao Co-authored-by: Ti Chi Robot * resolved_ts: fix shutdown panic (#12244) (#12245) close tikv/tikv#12231, ref tikv/tikv#12244 resolved_ts: fix shutdown panic Signed-off-by: ti-srebot Co-authored-by: NingLin-P * Quota limiter: make method to delay cpu more precise (#12219) (#12242) close tikv/tikv#12218, ref tikv/tikv#12219 Use the results returned by the cpu limiter directly without extra processing Signed-off-by: ti-srebot Co-authored-by: Xintao Co-authored-by: you06 * raftstore: validate bucket config only when it's enabled (#12237) (#12239) close tikv/tikv#12235, ref tikv/tikv#12236, ref tikv/tikv#12237 Shouldn't report config related errors when bucket is disabled. Signed-off-by: ti-srebot Co-authored-by: Lei Zhao Co-authored-by: Ti Chi Robot * res_meter: fix assertion failure (#12252) (#12261) close tikv/tikv#12234, ref tikv/tikv#12252 Signed-off-by: ti-srebot Co-authored-by: Yexiang Zhang * cdc: remove downstreams correctly when uninitialized (#12262) (#12268) ref tikv/tikv#12262 cdc: remove downstreams correctly when uninitialized Signed-off-by: ti-srebot Co-authored-by: qupeng * Update procinfo-rs to support pid status parse in kernel 5.x (#12263) (#12265) ref tikv/tikv#12263 Update procinfo-rs to support pid status parse in kernel 5.x. Signed-off-by: ti-srebot Co-authored-by: Xintao Co-authored-by: Ti Chi Robot * Quota limiter: increase refill duration to avoid much context switch cost (#12264) (#12266) ref tikv/tikv#12264 Quota limiter: increase refill duration to avoid much context switch cost. Signed-off-by: ti-srebot Co-authored-by: Xintao Co-authored-by: Ti Chi Robot * raftstore: only persist merge target if the merge is known to be succeeded (#12251) (#12271) close tikv/tikv#12232, ref tikv/tikv#12251 raftstore: only persist merge target if the merge is known to be succeeded Signed-off-by: ti-srebot Co-authored-by: NingLin-P Co-authored-by: you06 * storage: send old value to CDC after in-memory pessimistic locking (#12280) (#12282) close tikv/tikv#12279, ref tikv/tikv#12280 Send old values through the TxnExtraScheduler after in-memory pessimisitc locking successfully. This will help improve the old value hit rate of CDC when enabling in-memory pessimistic locks. Signed-off-by: ti-srebot Co-authored-by: Yilin Chen * server: fix issues with critical config check (#12250) (#12284) close tikv/tikv#12238, ref tikv/tikv#12238, ref tikv/tikv#12250 Signed-off-by: ti-srebot Co-authored-by: Xinye Tao * metrics: remove the incorrect rate() for worker pending tasks (#12152) (#12249) close tikv/tikv#11915, ref tikv/tikv#12152 tikv_worker_pending_task_total is a gauge. We should directly use its value to represent the pending tasks instead of using rate(). This PR removes the rate() on these metrics. Signed-off-by: ti-srebot Co-authored-by: Yilin Chen Co-authored-by: Ti Chi Robot * server: do not rename directories when migrating raft data (#12273) (#12298) close tikv/tikv#12269, ref tikv/tikv#12269, ref tikv/tikv#12273 Signed-off-by: ti-srebot Co-authored-by: Xinye Tao * *: update grpcio (#12203) (#12299) close tikv/tikv#12198, close tikv/tikv#12202, ref tikv/tikv#12203 update grpcio to fix UAF Signed-off-by: ti-srebot Co-authored-by: Jay * refine some versions Signed-off-by: CalvinNeo * add except exec_ctx Signed-off-by: CalvinNeo * use exec_log_index/term Signed-off-by: CalvinNeo * remove exec_ctc Signed-off-by: CalvinNeo * before handle cdc Signed-off-by: CalvinNeo * it compiles Signed-off-by: CalvinNeo * fmt Signed-off-by: CalvinNeo * Default disable raft engine (#12317) * disable raft engine Signed-off-by: tabokie * fix lint and test Signed-off-by: tabokie * bump version v6.0.0 Signed-off-by: purelind * fix linux Signed-off-by: CalvinNeo * address test deps Signed-off-by: CalvinNeo * Fix test Signed-off-by: CalvinNeo * f Signed-off-by: CalvinNeo * fix linux Signed-off-by: CalvinNeo * fix Signed-off-by: CalvinNeo * fix some files Signed-off-by: CalvinNeo * fix Signed-off-by: CalvinNeo * fix Signed-off-by: CalvinNeo * ignore some unsupported tests Signed-off-by: CalvinNeo * Fix test_delegate_subscribe_unsubscribe flaking test (#12309) (#12315) close tikv/tikv#12278, ref tikv/tikv#12309 Fix test_delegate_subscribe_unsubscribe flaking test. Use the corresponding id, otherwise there will be failures due to concurrent runs of the test. (We will assign downstream ids concurrently). Signed-off-by: ti-srebot Co-authored-by: 二手掉包工程师 Co-authored-by: qupeng Co-authored-by: Ti Chi Robot * fix fmt Signed-off-by: CalvinNeo * disable more disk_full tests Signed-off-by: CalvinNeo * f Signed-off-by: CalvinNeo * disable Signed-off-by: CalvinNeo * remove stale peer Signed-off-by: CalvinNeo * pick pr 12301 Signed-off-by: CalvinNeo * Fix build when openssl v1.1 and v3 are both installed (#59) Signed-off-by: Wish * Enable frame pointer (#60) Signed-off-by: mornyx * Upgrade pprof-rs to v0.9 (use frame-pointer), and enable CPU Profiling on ARM (#61) Signed-off-by: mornyx * reduce log Signed-off-by: CalvinNeo * raftstore:Fix append log entry oom (#12247) close tikv/tikv#11379, close tikv/tikv#11598, close tikv/tikv#12107 when memory usage is high water, add flow control on the append log entries. disable log entry cache evict functions. Signed-off-by: tier-cap Co-authored-by: Ti Chi Robot Signed-off-by: CalvinNeo * fix pd.rs of reporting to PD Signed-off-by: CalvinNeo * fix Signed-off-by: CalvinNeo * Raftstore proxy 6.0 try (#63) * raftstore: Introduce force leader state (#11932) close tikv/tikv#6107, ref tikv/tikv#10483 Signed-off-by: Connor1996 * raftstore: Wait ticks for hibernated peer when doing force leader (#12364) ref tikv/tikv#10483 Force leader is rejected on a peer who is already a leader. For the hibernated leader, it doesn't step down to follower when quorum is missing due to not ticking election. So wait ticks in force leader process for hibernated peers to make sure election ticking is performed. Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot * raftstore: Make unsafe recovery wait apply cover snapshot apply cases ref #10483 (#12308) ref tikv/tikv#10483 Signed-off-by: v01dstar * raftstore: Execute recovery plan via raft (#12022) Signed-off-by: Connor1996 Co-authored-by: Ti Chi Robot Co-authored-by: Yang Zhang * Hotfix pprof (#65) Signed-off-by: mornyx * remove log Signed-off-by: CalvinNeo * remove log Signed-off-by: CalvinNeo Co-authored-by: Qi Yu Co-authored-by: Xinye Tao Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta Co-authored-by: Ti Chi Robot Co-authored-by: Jack Yu Co-authored-by: xiongjiwei Co-authored-by: Ping Yu Co-authored-by: andylokandy Co-authored-by: qupeng Co-authored-by: Devillove084 Co-authored-by: Harold Dost Co-authored-by: zhangjinpeng1987 Co-authored-by: Yexiang Zhang Co-authored-by: 沐 Co-authored-by: Yilin Chen Co-authored-by: Zhenchi Co-authored-by: Harold Dost Co-authored-by: NingLin-P Co-authored-by: Peng Guanwen Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: Ryan Leung Co-authored-by: xufei Co-authored-by: 3pointer Co-authored-by: LemonHX Co-authored-by: you06 Co-authored-by: guo-shaoge Co-authored-by: Wenxuan Co-authored-by: Zwb Co-authored-by: Wenxuan Co-authored-by: Jay Co-authored-by: Liqi Geng Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com> Co-authored-by: Lei Zhao Co-authored-by: xiejiandong Co-authored-by: 5kbpers Co-authored-by: lhy1024 Co-authored-by: Yao Zongyou Co-authored-by: Connor Co-authored-by: tonyxuqqi Co-authored-by: qi.xu Co-authored-by: zhouqiang Co-authored-by: Xiang Zhang Co-authored-by: Mini256 Co-authored-by: Xiaoguang Sun Co-authored-by: Aarush Bhat Co-authored-by: Yifan Xu <30385241+xuyifangreeneyes@users.noreply.github.com> Co-authored-by: Suhaha Co-authored-by: goroutine Co-authored-by: Croxx Co-authored-by: zhongyong jin Co-authored-by: Mattias Jonsson Co-authored-by: JmPotato Co-authored-by: hi-rustin Co-authored-by: Song Gao Co-authored-by: Neil Shen Co-authored-by: Yang Zhang Co-authored-by: Jin Dong Co-authored-by: disksing Co-authored-by: 龙方淞 Co-authored-by: cui fliter Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: randy Co-authored-by: haojinming Co-authored-by: 虎 Co-authored-by: Alex Chi Co-authored-by: glorv Co-authored-by: Justin Huang Co-authored-by: Xintao Co-authored-by: haojinming Co-authored-by: cfzjywxk Co-authored-by: cosven Co-authored-by: Wallace Co-authored-by: ti-srebot <66930949+ti-srebot@users.noreply.github.com> Co-authored-by: YangKeao Co-authored-by: tier-cap Co-authored-by: purelind Co-authored-by: Wenxuan --- components/server/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/components/server/src/lib.rs b/components/server/src/lib.rs index 5b2f8a2da68..7c9352a8404 100644 --- a/components/server/src/lib.rs +++ b/components/server/src/lib.rs @@ -34,7 +34,7 @@ fn proxy_version_info() -> String { } fn log_proxy_info() { - info!("Welcome To RaftStore Proxy ZZZZZZZZZ"); + info!("Welcome To RaftStore Proxy"); for line in proxy_version_info().lines() { info!("{}", line); }