What is trained?

[jakobwes]

Hey everyone,

I have recently started using Jiant for my NLP-experiments and I really love the tool! I just have one question on the task-training and evaluation: when I run a task with a given model/encoder eg. BERT and with the respective head eg. the QAHead, if I do question answering, what is trained exactly? Is only the model-head trained, whilst the encoder remains the same, or does the loss calculated backpropagate further than the head, also in the layers of the encoder (so is the encoder modified during training)? I had a look at the complex_backpropagate-function and JiantModel-class, but am not sure.

If you have any hints that would be amazing. For my experiments I need to make sure that the encoder-layers remain the same, so it would be wonderful if this is done automatically or if not you have any advice on how to achieve that.

Best,
Jakob