元数据受损，恢复后3台FE数据无法同步，Observer节点无法Alive

[leiyangcd]

我三个FE，一个Follower，2个Observer，Follower节点因元数据受损挂掉，重启启动不起来，我从一个Observer节点进行数据恢复后，
启动后2个Observer节点无法Alive；fe.log中报错误日志如下：
com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 7.3.7) 192.168.x.x"_9010_1661853863401(2147483645):/doris/fe/doris-meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA 192.168.x.x"_9010_1661853863401(2147483645) Originally thrown by HA thread: REPLICA 192.168.x.x"_9010_1661853863401(2147483645)

[hf200012]

我三个FE，一个Follower，2个Observer，Follower节点因元数据受损挂掉，重启启动不起来，我从一个Observer节点进行数据恢复后， 启动后2个Observer节点无法Alive；fe.log中报错误日志如下： com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 7.3.7) 192.168.x.x"_9010_1661853863401(2147483645):/doris/fe/doris-meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA 192.168.x.x"_9010_1661853863401(2147483645) Originally thrown by HA thread: REPLICA 192.168.x.x"_9010_1661853863401(2147483645)

Add my WeChat: 35926237

[hf200012]

你也可以这样处理：

1. 在 fe.conf 添加配置：metadata_failure_recovery=true
2. 启动 FE ，如果正常
3. metadata_failure_recovery=false
4. 在重新启动 fe
5. 如果你是多台fe 的话，上面第4步执行成功之后，在mysql 命令行下，执行alter system drop follower
6. 将其他fe 从集群中删除
7. 然后再有问题的fe上，将元数据目录清空
8. 启动这个fe 启动命令里加 --helper 参数，这里helper 参数跟的是master fe的ip地址：端口（默认9010）
9. 在mysql 命令行下在把这节点重新加入到集群中即可
10. 其他的节点操作一样

[hf200012]

You can also handle it like this.

1. Add a configuration to fe.conf: metadata_failure_recovery=true
2. Start FE and if it works
3. metadata_failure_recovery=false
4. After restarting fe
5. If you have multiple FEs, after step 4 above is executed successfully, at the mysql command line, execute alter system drop follower
6. to remove the other fe from the cluster
7. Then empty the metadata directory on the problem fe
8. Start the fe and add the --helper parameter to the start command, where the helper parameter is the ip address of the master fe and the port (default 9010)
9. Re-add the node to the cluster at the mysql command line
10. Other nodes are the same