Contribute Hugging Face models to the MLX Community #155

awni · 2023-12-20T15:37:39Z

We encourage you to join the MLX Community on Hugging Face 🤗 and upload new MLX converted models and versions of existing models.

Vaibhavs10 · 2023-12-20T15:53:18Z

Love it! Here's some more information about uploading to the 🤗 hub.

Once you've converted the checkpoints via the convert.py files in the respective model directory, you can upload large files over to the hub as follows:

First, make sure to install huggingface_hub via pip install --upgrade huggingface_hub.

Second, run huggingface-cli login and paste a write token over from https://huggingface.co/settings/tokens.

Next, run the below code:

from huggingface_hub import HfApi
from huggingface_hub import logging

logging.set_verbosity_info()

api = HfApi()

api.upload_folder(folder_path="<LOCAL FOLDER PATH>", 
                  repo_id="mlx-community/<MODEL_NAME>", 
                  repo_type="model", 
                  multi_commits=True, 
                  multi_commits_verbose=True)

Read more here about file uploads here: https://huggingface.co/docs/huggingface_hub/guides/upload

maxtheman · 2023-12-20T20:22:49Z

Is there any more information about what's needed to author a convert.py for a given model? I'm seeing a lot of similarities between them in terms of loading the weights and then grouping into a .npz file but I'm not clear what's driving all of the differences.

Bert needs lots of keys replaced: https://github.com/ml-explore/mlx-examples/blob/main/bert/convert.py
Phi needs some: https://github.com/ml-explore/mlx-examples/blob/main/llms/phi2/convert.py
Mistral doesn't need any, very straightforward: https://github.com/ml-explore/mlx-examples/blob/main/llms/mistral/convert.py
and Mixtral has a bunch of stuff going on, but I think mostly for handling the MoE architecture.

In my case — I was just looking at converting Mamba to MLX. I think I have a good reference implementation for running the model but I'm having trouble understanding how I would convert the weights appropriately to test it. Looking at the repo on HF it's not obvious to me what I should be keying in on to start making decisions. https://huggingface.co/state-spaces/mamba-370m/tree/main

Any suggestions are appreciated.

awni · 2023-12-20T21:02:07Z

It's a bit more of an art at this point, we've been pushing for some standardization / consistency which is why things look the same. But there are still some subtle differences.

The short answer to your question is:

The output of convert.py should be two things:

weights.npz (or multiple weights files for large models)
config.json (to store the model metadata). Add a model_type field to this (for now it's unused, but will be important to work with Hugging Face in the future).

Beyond that it's really up to you how to structure the weights.npz and the config.json. You just need to be able to load them and construct the model in the file which actually runs the model. For the examples in mlx-examples, the weights.npz has keys which match the nn.Module hierarchy of the corresponding model. So you can basically do:

model.load_weights("weights.npz")

So to make that work cleanly, in the conversion code remap the keys in the PyTorch model state_dicts so they match the keys expected by the MLX model.

mzbac · 2023-12-21T09:31:18Z

@awni Thanks for this great initiative. I'm just curious why we can't provide the conversion script for people to convert the HF format model to mlx format? For example, the current conversion script for tiny_llama is pretty much compatible with the llama mode in HF format. What's the reason behind naming it as tinay_llama instead of something like hf_llama? https://github.com/ml-explore/mlx-examples/blob/main/llms/llama/convert.py#L54

awni · 2023-12-21T14:42:54Z

@mzbac that's a good question. Since different models have slight differences, we are doing a convention now where each model gets a converter. The underlying converters can use HF format (as in Tiny Llama) or original distributed format (as in Llama).

I know it's a bit convoluted at the moment. As we work with the community and you all we'll work towards simplifying and standardizing. Any suggestions and/or pinpoints on that front are appreciated.

PS the short term future I would like to move towards is to avoid a pre conversion step entirely:

Read the repo from HF in the safetensors format directly
Do any necessary remapping of keys in the weights and config on a model specific basis to make it all work

And someday MLX in Transformers directly, so no model.py file necessary either :)

mzbac · 2023-12-22T23:04:39Z

It would be better if we provided a script for downloading the models. That way, people wouldn't have to install git-lfs and pull the model via git.
Here is an example script I used in the past to pull the public model from Hugging Face.

import argparse
import requests
import os
from tqdm import tqdm

def download_file(url, path):
    response = requests.get(url, stream=True)
    total_size_in_bytes = int(response.headers.get('content-length', 0))
    block_size = 1024 #1 Kbyte
    progress_bar = tqdm(total=total_size_in_bytes, unit='iB', unit_scale=True)

    with open(path, 'wb') as file:
        for data in response.iter_content(block_size):
            progress_bar.update(len(data))
            file.write(data)
            
    progress_bar.close()

def download_model(model_name, destination_folder="models"):
    # Define the base URL and headers for the Hugging Face API
    base_url = f"https://huggingface.co/{model_name}/resolve/main"
    headers = {"User-Agent": "Hugging Face Python"}

    # Send a GET request to the Hugging Face API to get a list of all files
    response = requests.get(f"https://huggingface.co/api/models/{model_name}", headers=headers)
    response.raise_for_status()
    
    # Extract the list of files from the response JSON
    files_to_download = [file["rfilename"] for file in response.json()["siblings"]]

    # Ensure the directory exists
    os.makedirs(f"{destination_folder}/{model_name}", exist_ok=True)

    # Download each file
    for file in files_to_download:
        print(f"Downloading {file}...")
        download_file(f"{base_url}/{file}", f"{destination_folder}/{model_name}/{file}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("model_name", type=str, help="Name of the model to download.")
    args = parser.parse_args()

    download_model(args.model_name)

Just run

python download.py <huggingface_model_name> .eg. mlx-community/Llama-2-7b-chat-mlx

rupurt · 2024-01-03T18:35:34Z

@maxtheman I would also love to try an run mamba models with MLX. Did you end up finding any good resources?

Vaibhavs10 · 2024-01-04T09:17:31Z

Hi all, I hope you are doing well. With recently merged support for hf_llms, you should be able to use models directly from the Hugging Face Hub (Mistral and llama architectures are currently supported). 🔥

In addition to this we've shipped an updated conversion script - which allows you to provide a HF Hub model name and it automagically quantises and uploads the weights to mlx-community org.

You would need to do the following:

git clone https://github.com/ml-explore/mlx-examples
cd mlx-examples/llms/hf_llms
pip install -r requirements.txt
python convert. py --hf-path "codellama/CodeLlama-13b-Python-hf" -q --upload-name CodeLlama-13b-Python-hf-4bit-mlx

That's it!

Help us increase the coverage of MLX quantised checkpoints by quantising llama & mistral checkpoints from the hub.

Note: Currently, there is a minor bug in the conversion script, which errors out on upload. You can check out this PR (#221) for conversion.

Edit: The PR is now merged, so no need to worry about the Note above.

P.S. Feel free to tag @awni, @pcuenca or me (@Vaibhavs10) if you face any issues!

sukkritsharmaofficial · 2024-01-04T18:10:13Z

Hey @Vaibhavs10 , i'm getting this error while quantizing hf models to 4bit mlx :

self.weight, self.scales, self.biases = mx.quantize(weight, group_size, bits)
ValueError: [quantize] All dimensions should be divisible by 32 for now

conversion script :

python convert.py --hf-path "teknium/OpenHermes-2.5-Mistral-7B" -q --upload-name OpenHermes-2.5-Mistral-7B-hf-4bit-mlx

how do i fix this?

awni · 2024-01-04T20:15:31Z

@sukkritsharmaofficial check the issue here for the reason.

Looking at that model I think it's possible the issue is the vocab size is not divisible by 32 (I see in the config "vocab_size": 32002). That's a tricky one. It may need to wait for non-32 multiples.

CC @angeloskath

tingbaozhao · 2024-02-01T08:03:09Z

Hi all, I hope you are doing well. With recently merged support for hf_llms, you should be able to use models directly from the Hugging Face Hub (Mistral and llama architectures are currently supported). 🔥

In addition to this we've shipped an updated conversion script - which allows you to provide a HF Hub model name and it automagically quantises and uploads the weights to mlx-community org.

You would need to do the following:

git clone https://github.com/ml-explore/mlx-examples

cd mlx-examples/llms/hf_llms

pip install -r requirements.txt

python convert. py --hf-path "codellama/CodeLlama-13b-Python-hf" -q --upload-name CodeLlama-13b-Python-hf-4bit-mlx

That's it!

Help us increase the coverage of MLX quantised checkpoints by quantising llama & mistral checkpoints from the hub.

Note: Currently, there is a minor bug in the conversion script, which errors out on upload. You can check out this PR (#221) for conversion.

Edit: The PR is now merged, so no need to worry about the Note above.

P.S. Feel free to tag @awni, @pcuenca or me (@Vaibhavs10) if you face any issues!

the link is not exists ? click hf_llms, i got a 404 error !

mzbac · 2024-02-01T08:14:33Z

Hi all, I hope you are doing well. With recently merged support for hf_llms, you should be able to use models directly from the Hugging Face Hub (Mistral and llama architectures are currently supported). 🔥
In addition to this we've shipped an updated conversion script - which allows you to provide a HF Hub model name and it automagically quantises and uploads the weights to mlx-community org.
You would need to do the following:

git clone https://github.com/ml-explore/mlx-examples

cd mlx-examples/llms/hf_llms

pip install -r requirements.txt

python convert. py --hf-path "codellama/CodeLlama-13b-Python-hf" -q --upload-name CodeLlama-13b-Python-hf-4bit-mlx

That's it!
Help us increase the coverage of MLX quantised checkpoints by quantising llama & mistral checkpoints from the hub.
Note: Currently, there is a minor bug in the conversion script, which errors out on upload. You can check out this PR (#221) for conversion.
Edit: The PR is now merged, so no need to worry about the Note above.
P.S. Feel free to tag @awni, @pcuenca or me (@Vaibhavs10) if you face any issues!

the link is not exists ? click hf_llms, i got a 404 error !

It has been moved to https://github.com/ml-explore/mlx-examples/tree/main/llms :)

Anirud-Mohan · 2024-02-15T05:11:31Z

hey @awni , I'm trying to fuse my finetuned model into a GGUF file. But while I execute the fuse command, I'm getting list index out of range error. The fuse command I used is as follows :
python fuse.py --model mistralai/Mistral-7B-Instruct-v0.2 --save-path ./lora-fused-model --adapter-file adapters.npz --de-quantize

The error thrown is as follows :
File "/Users/tis-ai-server/workspace/mlx-examples/.venv/lib/python3.11/site-packages/mlx/utils.py", line 123, in tree_unflatten int(tree[0][0].split(".", maxsplit=1)[0]) ~~~~^^^ IndexError: list index out of range

awni · 2024-02-15T05:25:44Z

That is strange. Can you share the full stack trace so we can see more precisely the issue arises?

Anirud-Mohan · 2024-02-15T06:38:06Z

@awni Here's the full stack trace, hope this helps :

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. Loading pretrained model Fetching 11 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 109071.74it/s] Traceback (most recent call last): File "/Users/tis-ai-server/workspace/mlx-examples/lora/fuse.py", line 98, in <module> model.update_modules(tree_unflatten(de_quantize_layers)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/tis-ai-server/workspace/mlx-examples/.venv/lib/python3.11/site-packages/mlx/utils.py", line 123, in tree_unflatten int(tree[0][0].split(".", maxsplit=1)[0]) ~~~~^^^ IndexError: list index out of range

sukhvir1313 · 2024-03-23T14:32:05Z

@awni Here's the full stack trace, hope this helps :

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. Loading pretrained model Fetching 11 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 109071.74it/s] Traceback (most recent call last): File "/Users/tis-ai-server/workspace/mlx-examples/lora/fuse.py", line 98, in <module> model.update_modules(tree_unflatten(de_quantize_layers)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/tis-ai-server/workspace/mlx-examples/.venv/lib/python3.11/site-packages/mlx/utils.py", line 123, in tree_unflatten int(tree[0][0].split(".", maxsplit=1)[0]) ~~~~^^^ IndexError: list index out of range

Any updates on this error, I am also receiving this kind of error.

awni · 2024-03-23T14:33:19Z

Can you share steps to reproduce?

sukhvir1313 · 2024-03-24T03:26:34Z

Can you share steps to reproduce?

Below is the command and result that I got.

`conda activate AppleMLXExample

(AppleMLXExample) usermlx@MacBook-Pro lora % python fuse.py --model mistralai/Mistral-7B-Instruct-v0.2 --adapter-file adapters.npz --save-path Models/My-Mistral-7B-fine-tuned --upload-name My-Mistral-7B-fine-tuned --de-quantize
Loading pretrained model
Fetching 11 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 111983.84it/s]
Traceback (most recent call last):
File "/Users/usermlx/Documents/Me/programming/ai/mlx-examples/lora/fuse.py", line 98, in
model.update_modules(tree_unflatten(de_quantize_layers))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/usermlx/miniconda3/envs/AppleMLXExample/lib/python3.11/site-packages/mlx/utils.py", line 122, in tree_unflatten
int(tree[0][0].split(".", maxsplit=1)[0])
~~~~^^^
IndexError: list index out of range`

mzbac · 2024-03-24T03:33:46Z

Can you share steps to reproduce?

Below is the command and result that I got.

`conda activate AppleMLXExample

(AppleMLXExample) usermlx@MacBook-Pro lora % python fuse.py --model mistralai/Mistral-7B-Instruct-v0.2 --adapter-file adapters.npz --save-path Models/My-Mistral-7B-fine-tuned --upload-name My-Mistral-7B-fine-tuned --de-quantize Loading pretrained model Fetching 11 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 111983.84it/s] Traceback (most recent call last): File "/Users/usermlx/Documents/Me/programming/ai/mlx-examples/lora/fuse.py", line 98, in model.update_modules(tree_unflatten(de_quantize_layers)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/usermlx/miniconda3/envs/AppleMLXExample/lib/python3.11/site-packages/mlx/utils.py", line 122, in tree_unflatten int(tree[0][0].split(".", maxsplit=1)[0]) ~~~~^^^ IndexError: list index out of range`

It seems like this was caused by trying to de-quantize a non-quantized model. Would you try running the fuse without the --de-quantize flag?

sukhvir1313 · 2024-03-24T06:14:43Z

Yes, it works that way.Sent from my iPhoneOn 24 Mar 2024, at 9:04 AM, Anchen ***@***.***> wrote: Can you share steps to reproduce? Below is the command and result that I got. `conda activate AppleMLXExample (AppleMLXExample) ***@***.*** lora % python fuse.py --model mistralai/Mistral-7B-Instruct-v0.2 --adapter-file adapters.npz --save-path Models/My-Mistral-7B-fine-tuned --upload-name My-Mistral-7B-fine-tuned --de-quantize Loading pretrained model Fetching 11 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 111983.84it/s] Traceback (most recent call last): File "/Users/usermlx/Documents/Me/programming/ai/mlx-examples/lora/fuse.py", line 98, in model.update_modules(tree_unflatten(de_quantize_layers)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/usermlx/miniconda3/envs/AppleMLXExample/lib/python3.11/site-packages/mlx/utils.py", line 122, in tree_unflatten int(tree[0][0].split(".", maxsplit=1)[0]) ~~~~^^^ IndexError: list index out of range` It seems like this was caused by trying to de-quantize a non-quantized model. Would you try running the fuse without the --de-quantize flag? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

awni pinned this issue Dec 20, 2023

dastrobu mentioned this issue Dec 22, 2023

Llama/unshard on load #174

Merged

gboduljak mentioned this issue Jan 14, 2024

[Feature Request] Example of MLLM using MLX #207

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Contribute Hugging Face models to the MLX Community #155

Contribute Hugging Face models to the MLX Community #155

awni commented Dec 20, 2023 •

edited

Loading

Vaibhavs10 commented Dec 20, 2023 •

edited

Loading

maxtheman commented Dec 20, 2023

awni commented Dec 20, 2023

mzbac commented Dec 21, 2023 •

edited

Loading

awni commented Dec 21, 2023 •

edited

Loading

mzbac commented Dec 22, 2023

rupurt commented Jan 3, 2024

Vaibhavs10 commented Jan 4, 2024 •

edited

Loading

sukkritsharmaofficial commented Jan 4, 2024 •

edited

Loading

awni commented Jan 4, 2024

tingbaozhao commented Feb 1, 2024 •

edited

Loading

mzbac commented Feb 1, 2024

Anirud-Mohan commented Feb 15, 2024

awni commented Feb 15, 2024

Anirud-Mohan commented Feb 15, 2024

sukhvir1313 commented Mar 23, 2024

awni commented Mar 23, 2024

sukhvir1313 commented Mar 24, 2024 •

edited

Loading

mzbac commented Mar 24, 2024

sukhvir1313 commented Mar 24, 2024 via email

Contribute Hugging Face models to the MLX Community #155

Contribute Hugging Face models to the MLX Community #155

Comments

awni commented Dec 20, 2023 • edited Loading

Vaibhavs10 commented Dec 20, 2023 • edited Loading

maxtheman commented Dec 20, 2023

awni commented Dec 20, 2023

mzbac commented Dec 21, 2023 • edited Loading

awni commented Dec 21, 2023 • edited Loading

mzbac commented Dec 22, 2023

rupurt commented Jan 3, 2024

Vaibhavs10 commented Jan 4, 2024 • edited Loading

sukkritsharmaofficial commented Jan 4, 2024 • edited Loading

awni commented Jan 4, 2024

tingbaozhao commented Feb 1, 2024 • edited Loading

mzbac commented Feb 1, 2024

Anirud-Mohan commented Feb 15, 2024

awni commented Feb 15, 2024

Anirud-Mohan commented Feb 15, 2024

sukhvir1313 commented Mar 23, 2024

awni commented Mar 23, 2024

sukhvir1313 commented Mar 24, 2024 • edited Loading

mzbac commented Mar 24, 2024

sukhvir1313 commented Mar 24, 2024 via email

awni commented Dec 20, 2023 •

edited

Loading

Vaibhavs10 commented Dec 20, 2023 •

edited

Loading

mzbac commented Dec 21, 2023 •

edited

Loading

awni commented Dec 21, 2023 •

edited

Loading

Vaibhavs10 commented Jan 4, 2024 •

edited

Loading

sukkritsharmaofficial commented Jan 4, 2024 •

edited

Loading

tingbaozhao commented Feb 1, 2024 •

edited

Loading

sukhvir1313 commented Mar 24, 2024 •

edited

Loading