Cannot decommission BE node from current clusters

[Li-Yanzhi]

Environment Summary

1. Setup a Doris (v0.15) cluster on 3 Ubuntu servers, each server have both FE & BE role.
2. Create sample db via benchmark script (https://doris.apache.org/master/en/benchmark/star-schema-benchmark.html)
3. Add 3 new Ubuntu servers to cluster, each server have both FE & BE role.

Requirement

To migrate workload/data from origin 3 servers to 3 new servers, then retire & delete 3 old servers.

Issue

I have successfully drop FE roles from 3 old servers, but when I decommssion the first server of 3 old servers, then I found then TabletNum on this will remain a fixed value and won't decrease for days, so I cannot drop and retire the old server one by onesafely.

The command and result is as bellow:

mysql> show backends;

+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+
| BackendId | Cluster         | IP        | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | Tag                      | ErrMsg | Version       | Status                                                                                                                        |
+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+
| 10002     | default_cluster | 10.3.2.11 | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:37:20 | 2021-12-23 12:09:45 | true  | false                | false                 | 19        | 8.008 MB         | 84.570 GB     | 96.724 GB     | 12.56 % | 12.56 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:09:27","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10003     | default_cluster | 10.3.2.12 | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:37:59 | 2021-12-23 12:09:45 | true  | false                | false                 | 34        | 13.874 MB        | 86.759 GB     | 96.724 GB     | 10.30 % | 10.30 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:08:44","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10004     | default_cluster | 10.3.2.13 | 9050          | 9060   | 8040     | 8060     | 2021-12-15 07:40:52 | 2021-12-23 12:09:45 | true  | false                | false                 | 33        | 13.457 MB        | 86.513 GB     | 96.724 GB     | 10.56 % | 10.56 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:08:56","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10798     | default_cluster | 10.3.1.5  | 9050          | 9060   | 8040     | 8060     | 2021-12-15 08:08:35 | 2021-12-23 12:09:45 | true  | false                | false                 | 92        | 36.586 MB        | 111.018 GB    | 116.129 GB    | 4.40 %  | 4.40 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:08:54","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10949     | default_cluster | 10.3.1.6  | 9050          | 9060   | 8040     | 8060     | 2021-12-15 09:15:50 | 2021-12-23 12:09:45 | true  | false                | false                 | 92        | 34.100 MB        | 120.635 GB    | 125.821 GB    | 4.12 %  | 4.12 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:08:58","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10950     | default_cluster | 10.3.1.7  | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:30:26 | 2021-12-23 12:09:45 | true  | false                | false                 | 97        | 40.207 MB        | 120.929 GB    | 125.821 GB    | 3.89 %  | 3.89 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:09:04","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+

*** Notes:  Current status: node 1 (10.3.2.11) have 19 TabletNum and working properly

mysql> show proc "/statistic";
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
| DbId  | DbName              | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | BadTabletNum |
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
| 10005 | default_cluster:db1 | 5        | 11           | 11       | 367       | 367        | 0                  | 0                     | 0                | 0            |
| Total | 1                   | 5        | 11           | 11       | 367       | 367        | 0                  | 0                     | 0                | 0            |
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
2 rows in set (0.01 sec)

*** Notes:  There is no any UnhealthyTabletNum 

mysql> ALTER SYSTEM DECOMMISSION BACKEND "10.3.2.11:9050";
Query OK, 0 rows affected (0.02 sec)

*** Notes:  Decommssion node 1 (10.3.2.11) from cluster

mysql> show backends;
+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+
| BackendId | Cluster         | IP        | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | Tag                      | ErrMsg | Version       | Status                                                                                                                        |
+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+
| 10002     | default_cluster | 10.3.2.11 | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:37:20 | 2021-12-23 12:12:46 | true  | true                 | false                 | 19        | 8.008 MB         | 84.570 GB     | 96.724 GB     | 12.56 % | 12.56 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:12:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10003     | default_cluster | 10.3.2.12 | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:37:59 | 2021-12-23 12:12:46 | true  | false                | false                 | 34        | 13.874 MB        | 86.759 GB     | 96.724 GB     | 10.30 % | 10.30 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:11:56","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10004     | default_cluster | 10.3.2.13 | 9050          | 9060   | 8040     | 8060     | 2021-12-15 07:40:52 | 2021-12-23 12:12:46 | true  | false                | false                 | 33        | 13.457 MB        | 86.513 GB     | 96.724 GB     | 10.56 % | 10.56 %        | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:12:03","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10798     | default_cluster | 10.3.1.5  | 9050          | 9060   | 8040     | 8060     | 2021-12-15 08:08:35 | 2021-12-23 12:12:46 | true  | false                | false                 | 92        | 36.586 MB        | 111.018 GB    | 116.129 GB    | 4.40 %  | 4.40 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:12:04","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10949     | default_cluster | 10.3.1.6  | 9050          | 9060   | 8040     | 8060     | 2021-12-15 09:15:50 | 2021-12-23 12:12:46 | true  | false                | false                 | 92        | 34.100 MB        | 120.635 GB    | 125.821 GB    | 4.12 %  | 4.12 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:12:08","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
| 10950     | default_cluster | 10.3.1.7  | 9050          | 9060   | 8040     | 8060     | 2021-12-23 09:30:26 | 2021-12-23 12:12:46 | true  | false                | false                 | 97        | 40.207 MB        | 120.929 GB    | 125.821 GB    | 3.89 %  | 3.89 %         | {"location" : "default"} |        | trunk-2ae9c41 | {"lastSuccessReportTabletsTime":"2021-12-23 12:12:10","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |
+-----------+-----------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+---------------+-------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.04 sec)

*** Notes:  Node 1 (10.3.2.11) SystemDecommissioned status changed from false to true, it means the node is decommssioning 

mysql> show proc "/statistic";
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
| DbId  | DbName              | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | BadTabletNum |
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
| 10005 | default_cluster:db1 | 5        | 11           | 11       | 367       | 367        | 19                 | 0                     | 0                | 0            |
| Total | 1                   | 5        | 11           | 11       | 367       | 367        | 19                 | 0                     | 0                | 0            |
+-------+---------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+--------------+
2 rows in set (0.00 sec)

*** Notes:  There are 19 TabletNum is Unhealthy, just as same as TabletNum on Node 1 (10.3.2.11) , this status have remain for several days  and node 1 (10.3.2.11) status also keep no change too.

Does anyone have experiences on this? or give me some advices to further debug this issue. Thanks in advance

[morningman]

You need to use show proc "/cluster_balance/" to see if there are running of pending clone tasks.

[Li-Yanzhi]

Thanks @morningman for debugging & fixing this issue, link: #7563