Sequential Fine-Tuning of Pretrained Hugging Face Models for Improved Object Detection Accuracy

[abeyankargiridharan]

The proposed implementation involves a two-stage sequential finetuning process to improve the accuracy of pretrained huggingface models for object detection.

• The initial stage involves finetuning the existing pretrained hf model on the first dataset (Fisheye 8k) to improve feature extraction under fisheye imaging conditions.
• The next stage incorporates further finetuning on the other dataset (Mcity Fisheye ) to enhance the detection accuracy by adjusting the weights of specific layers .

To address the differences in class labels between the two datasets, the intelligent class mapping workflow will be leveraged. This workflow uses zero-shot image classification models to perform one-to-many mapping for detections, ensuring alignment between the labels of both datasets.

The final model, trained on both datasets, will be evaluated using metrics such as F1 score, precision, and recall to assess the magnitude of improvement as a result of sequential fine-tuning.