Code for "Memory-Tuning"
Our released code is setted for using BERT-Large. If you want to validate results conducted on BERT-base, Please revise code as the following instruction:
Step1: In ffn_trainabl e_module/bert.py line 120-121, set n_layer=12 n_head=12, n_embd=768, mid_dim=1024
Step2: In main.py line 77, set hidden_size=768
script.sh supports 7 tuning methods for training and inferring on 8 dataset used in paper. There are line command examples running on 3 datasets in given script.sh. You can add line commands for other datasets according to those given ones and please note different setup for different dataset in Table 1 of our paper.
Our released code is setted for using BERT-Large but our visualization experiment is conducted by using BERT-base. So you need to revise several places in code for visualization experiment as the Note part illustrated.
Step1: revise code as Note part illustrated.
Step2: Train BERT-base on SST-2 and MRPC for MT-M, MT-F, MT-MF.
Step3: use Visualize/MSRA/bert.py to replace Token-Level/ffn_trainable_module/bert.py for MSRA or use Visualize/SST-2/bert.py to replace Sentence-Level/ffn_trainable_module/bert.py for SST-2
Step4: use predict.py for inferring
Take visualization of memory using MT-F for example, MT-M and MT-MF are similar with it in the following steps.
Step5: set --test-file './data/SST-2/dev.tsv' --test-file './data/MRPC/dev.tsv' in line command for SST-2 and MRPC respectively. (Since we experiment on the dev file)
Step6: copy the inferred attention data and pred.txt into Visualize\MSRA\memory-ffn, Run compare.txt and then Run t-sne.py
If you have any question, feel free to connect me by sending an email into [email protected]. :)