refactor(yaml): Config ctc/cmvn/tokenizer in train.yaml

[xingchensong]

1. 将模型相关的配置从train args移动到yaml，比如ctc、cmvn、tokenizer
2. 将sos设定为恒定=2，而不是vocab_size - 1,特殊化处理，此时不同字典可以share相同的特殊token

TODO (current PR)

• 验证可以训练
• 验证可以解码

TODO (next PR)

• librispeech 适配 [recipe] refine yaml for librispeech #2227
• wenetspeech 适配 [recipe] refine yaml for wenetspeech #2229
• gigaspeech 适配 [recipe] refine yaml for gigaspeech #2230
• multi_cn 适配
• aishell2 适配 [recipe] refine yaml for aishell2 #2231
• pretrained model 重新上传 [doc] update aishell url #2221
• paraformer相关修改 @Mddct [paraformer] refine model class #2226

[Mddct]

great work

[Mddct]

TODO

• paraformer special tokens @Mddct refactor(yaml): Config ctc/cmvn/tokenizer in train.yaml #2205 (comment)
• tokenizer check special tokens valid @Mddct

[robin1001]

周哥，这个改动比较大，功能也很杂，强烈建议分开提 PR，每个 PR 只做一个功能，来自 Google 软件实践指南中的建议。

[xingchensong]

记一个TODO，后面runtime也要针对special id做对应的修改，比如blanckid不是0，sos不是vocabsize-1等

[xingchensong]

模型转换脚本（sos从vocabsize-1变成2）

import torch                                                                                                                                                                                                                  [3/528]

old_state = torch.load('/mnt/d/BaiduSyncdisk/downloads/ckpt/20210601_u2++_conformer_exp_aishell/final.pt')
new_state = {}
change_list = ['decoder.left_decoder.output_layer.weight',
               'decoder.left_decoder.output_layer.bias',
               'decoder.left_decoder.embed.0.weight',
               'decoder.right_decoder.output_layer.weight',
               'decoder.right_decoder.output_layer.bias',
               'decoder.right_decoder.embed.0.weight',
               'ctc.ctc_lo.weight',
               'ctc.ctc_lo.bias']
for key in old_state.keys():
    if key in change_list:
        print("processing {}, {}".format(key, old_state[key].size()))
        tensor = old_state[key]
        new_tensor = torch.zeros_like(tensor)
        if len(tensor.size()) == 2:  # weight
            new_tensor[:2, :] = tensor[:2, :]
            new_tensor[2, :] = tensor[-1, :]
            new_tensor[3:, :] = tensor[2:-1, :]
        elif len(tensor.size()) == 1:  # bias
            new_tensor[:2] = tensor[:2]
            new_tensor[2] = tensor[-1]
            new_tensor[3:] = tensor[2:-1]
        else:
            raise NotImplementedError
        new_state[key] = new_tensor
    elif "concat_linear" in key:
        continue
    else:
        new_state[key] = old_state[key]

torch.save(new_state, "/mnt/d/BaiduSyncdisk/downloads/ckpt/20210601_u2++_conformer_exp_aishell/final.sos2.pt")

转换后可以成功解码

[xingchensong]

解码结果一致