FarmReader issue when loading fine-tuned and saved model

[KarlosMuradyan]

Hi,

I'm trying to load locally saved Roberta model that was fine-tuned and saved using FarmReader:

from haystack.nodes import FARMReader

top_k_reader = 1
squad_destination_train = 'squad.json'
squad_destination_valid = 'squad.json'

reader = FARMReader("deepset/roberta-base-squad2", top_k=top_k_reader, return_no_answer=False)
reader.train(data_dir='.', use_gpu=True, n_epochs=2, batch_size=2,
             train_filename=squad_destination_train, dev_filename=squad_destination_valid,
             save_dir="finetuned_models/roberta/finetuned_v1")

I tried using reader.save(...) as well but no difference in result. Here is how I want to load the saved model:

new_reader = FARMReader(model_name_or_path="finetuned_models/roberta/finetuned_v1")

I get the following error:

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'finetuned_models/roberta/finetuned_v1/language_model.bin'. Use `repo_type` argument if needed.

Based on the Reader documentation the above code should work. Am I missing something?

Thanks in advance!

Farm Haystack version:
farm-haystack==1.23.0

[vblagoje]

Hey, @KarlosMuradyan, this is today's second report about a similar issue. My guess is that the transformers library introduced a bug where they first don't check the local path or they have completely revamped how local models are loaded. I'll take a look tomorrow but if you do before that, please let us know. Apologies for the delay in responding 🙏

[KarlosMuradyan]

Hi @vblagoje
I recreated my environment and updated transformers library to version 4.37.0. Now it works but throws a warning when initializing FARMReader:
new_reader = FARMReader(model_name_or_path="finetuned_models/roberta/finetuned_v1", top_k=top_k_reader, return_no_answer=False)

Some unused parameters are passed to the QuestionAnsweringHead. Might not be a problem. Params: {"training": false, "num_labels": 2, "ph_output_type": "per_token_squad", "model_type": "span_classification", "label_tensor_name": "question_answering_label_ids", "label_list": ["start_token", "end_token"], "metric": "squad", "name": "QuestionAnsweringHead"}

My guess is that this is expected? The evaluation results seem to be matching with what the fine-tuned model gave without saving and loading.