Add PaliGemma LoRA

[probicheaux]

Description

Add in class that can perform inference using LoRAs

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

Locally

Any specific deployment considerations

n/a

Docs

• Docs updated? What were the changes:

[probicheaux]

1. PaliGemma needs transformers>=4.41.1, but requirements.cogvlm.txt and reqiurements.groundingdino.txt pin transformers low. Can we avoid that now?
2. This change doesn't work with get_model because we don't know a priori if the PaliGemma model is a LoRA or not. How should I handle that? Put something in the model bucket and check for that? Right now, there's a file adapter_config.json that exists if and only if the model is a LoRA. Should I use that file to check which class to load in get_model?

[PawelPeczek-Roboflow]

I remember pinning the version due to: #355
Not sure if never versions of transformers solve the problem, but if yes, probably lower-bound should be bumped

[PawelPeczek-Roboflow]

I know that not part of the change - but is that needed?

[probicheaux]

sorry, is what needed? the main? no, it was just for testing, we can remove

[PawelPeczek-Roboflow]

yeah, main

[PawelPeczek-Roboflow]

Could you post a docs example + description on LoRA model?
I guess this class is needed to be able to load LoRA-fine-tuned models from hf hub as people are posting those, but what about our platform? PaliGemma is RoboflowInferenceModel, but it seems that we don't load weights from our hosting - which may be the indication that this is kind of "core" model?
Also - is that always required to have HF token - or maybe we could rely on their auth?

[probicheaux]

I can write up something more detailed but the short answer is this

1. LoRA is a technique to train a small "diff" from some base model A
2. This PR assumes that users will deploy the LoRA (the diff) to Roboflow, but in order to use it, they will need to download the base model A from huggingface
3. This doesn't reduce the amount of data transferred on the first LoRA loaded, but will significantly reduce data transfer on subsequent LoRA loads -- from 6GB for a fully finetuned model, to only 28MB for a new LoRA
4. This will also reduce our storage needs because we don't need to host 6GB for each fine tune, just 28MB for each LoRA

[probicheaux]

So to be clear, we are loading weights from our hosting

[probicheaux]

Also - is that always required to have HF token - or maybe we could rely on their auth?

I'm not sure I understand this question

[PawelPeczek-Roboflow]

from previous answers I see that HF tokens will be required, at least sometimes

[capjamesg]

I have tested this implementation and successfully trained a model with LoRA.

[PawelPeczek-Roboflow]

Fine, as long as we resolve this #464 (comment) we are free to merge, I believe that would only take testing CogVLM and probably setting transformers>=4.41.1

[PawelPeczek-Roboflow]

Regarding question 2 form here
get_model(...) calls internally get_model_type(...). Would be best if we could have the information responded from API at that level.
If not feasible, relying on adapter_config.json is ok, but that probably would take having a single class for LoRA and non-LoRA versions?

[PawelPeczek-Roboflow]

@probicheaux - how we plan to move on with this?

[probicheaux]

@PawelPeczek-Roboflow sorry, I've been super busy. Just fixed the get_model thing by pushing a new model_conversion param that adds peft to lora models. I also tested cogvlm in the new docker container (verifying transformers==4.41.2 and it works fine.