Fine-tuning a custom BERT model

[Avs-safety]

Hello,

I'm looking to fine-tune my own BERT model on Q&A using haystack (note - this is a locally trained BERT model and not from huggingface), however, when running;

reader.train(data_dir=data_dir, train_filename="/gpfs/home/answers.json", use_gpu=True, n_epochs=10, save_dir="/gpfs/home/haystack_bert_reader3")

I get the following error:

2023-08-18 11:06:23.684191: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-18 11:06:24.917363: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-08-18 11:06:24.917523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-08-18 11:06:24.917545: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()

Preprocessing dataset:   0%|          | 0/1 [00:00<?, ? Dicts/s]
Preprocessing dataset:   0%|          | 0/1 [00:01<?, ? Dicts/s]
Traceback (most recent call last):
  File "/gpfs/home/haystack_reader.py", line 21, in <module>
    reader.train(data_dir=data_dir, train_filename="/gpfs/home/answers.json", use_gpu=True, n_epochs=10, save_dir="/gpfs/home/haystack_bert_reader3")
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/nodes/reader/farm.py", line 452, in train
    return self._training_procedure(
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/nodes/reader/farm.py", line 285, in _training_procedure
    data_silo = DataSilo(
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/data_silo.py", line 104, in __init__
    self._load_data()
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/data_silo.py", line 165, in _load_data
    self.data["train"], self.tensor_names = self._get_dataset(train_file)
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/data_silo.py", line 125, in _get_dataset
    dataset, tensor_names, problematic_sample_ids = self.processor.dataset_from_dicts(
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/processor.py", line 488, in dataset_from_dicts
    dataset, tensor_names, baskets = self._create_dataset(baskets)
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/processor.py", line 834, in _create_dataset
    dataset, tensor_names = convert_features_to_dataset(features=features_flat)
  File "/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/site-packages/haystack/modeling/data_handler/dataset.py", line 74, in convert_features_to_dataset
    cur_tensor = torch.as_tensor(np.array([sample[t_name] for sample in features]), dtype=torch.long)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4587,) + inhomogeneous part.
/gpfs/home/.conda/envs/JR-PyTorch-2.0-GPU-Python-3.9/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/tmp/pbs.47528.cmrc-pbs01/tmper1d_zhl'>
  _warnings.warn(warn_message, ResourceWarning)

Does this happen because of how my local model was created? Grateful for any help!

[bogdankostic]

Hi @Avs-safety! Without having access to your local model and knowing the details about it, it's hard to say what causes your issue. Is your local model in the same format as the models from huggingface transformers?

[Avs-safety]

I believe it is, the image below is a screenshot of the model directory

[bogdankostic]

Do you think you could share the model with me? Otherwise, it's really difficult to figure out what's going wrong.

[Avs-safety]

Thanks @bogdankostic I'm pretty sure I've resolved this now as didn't specify my training data in the correct place 🤦‍♂️

[Avs-safety]

Hi @bogdankostic , is it still possible to share one of my models with you please? I have the same issue, the training data is OK as I can train deepset/bert-cased-squad2, however when I come to train one of my models I get the value error as shown above. Wondering if the model is missing a config setting... Thanks!