Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add llamafile code snippet #1088

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

add llamafile code snippet #1088

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d61977d
add llamafile code snippet
not-lain Jan 5, 2025
803a605
update comment
not-lain Jan 5, 2025
e905022
reset gguf
not-lain Jan 5, 2025
58a2255
reset gguf
not-lain Jan 5, 2025
904599f
refactor llamafile implementation
not-lain Jan 5, 2025
9f06b2c
Update packages/tasks/src/local-apps.ts
not-lain Jan 7, 2025
e6ac3fb
Merge branch 'main' into add-llamafile-code-snippet
pcuenca Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions packages/tasks/src/local-apps.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ function isLlamaCppGgufModel(model: ModelData) {
return !!model.gguf?.context_length;
}

function isLlamaFileModel(model: ModelData){
not-lain marked this conversation as resolved.
Show resolved Hide resolved
return model.tags.includes("llamafile")
}

function isMlxModel(model: ModelData) {
return model.tags.includes("mlx");
}
Expand Down Expand Up @@ -137,6 +141,37 @@ const snippetLlamacpp = (model: ModelData, filepath?: string): LocalAppSnippet[]
];
};

const snippetLlamafile = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
const LinuxAndMacCommand = () => {
const snippet = [
"# Load and run the model:",
`wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? "{{LLMAFILE_FILE}}"}`,
Copy link
Member

@pcuenca pcuenca Jan 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is LLMAFILE_FILE correct? Should it be LLAMAFILE_FILE? Where is it defined?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for raising this @pcuenca
this was the most confusing aspect of this PR, i found {{GGUF_FILE}} as well but it was not defined anywhere else in the repo, would love to get some inputs on this one.
maybe it's some kind of a jinja pattern of some sorts, if so let me know what other changes that needs to be made.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mishig25 will be able to help

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplest would be to get one example file ending with .llamafile (if we have the list of files here, i don't remember) – not sure we need to implement the same selector as for GGUF @mishig25 – maybe overkill here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for 🦙📁 the file needs to have .llamafile somewhere in the filename but it might not need to end with the file_extension.
an example of this might be https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile/tree/main

for heavy files you might find that the cat[number] might be either at the end or right before the file_extension.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, I didn't realize we were planning to use a similar method as for gguf, makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at llamafile model repos on the hub: https://huggingface.co/models?library=llamafile, I only see a few from Mozilla which have more than 2-3 llama files.

Unless implementing a selector is trivial, I'd recommend that we pick up the first file/ or just name xyz.llamafile as we did in really early GGUF snippets.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes^

(BTW we only detect *.llamafile files as library=llamafile in the Hub. That's fine IMO, let's keep things simple)

`chmod +x ${filepath ?? "{{LLMAFILE_FILE}}"}`,
`./${filepath ?? "{{LLMAFILE_FILE}}"} -p "Once upon a time,"`,
];
return snippet.join("\n");
};
const WindowsCommand = () => {
const BaseFilename = (filepath ?? "{{LLMAFILE_FILE}}").split('/').pop()?.replace('.llamafile', '.exe');
const snippet = [
"# Load and run the model:",
`wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? "{{LLMAFILE_FILE}}"} -O ${BaseFilename}`,
`./${BaseFilename} -p "Once upon a time,"`,
];
return snippet.join("\n");
};
return [
{
title: "Linux and MacOS",
content: LinuxAndMacCommand(),
},
{
title: "Windows",
content: WindowsCommand(),
},
];
};

const snippetNodeLlamaCppCli = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
return [
{
Expand Down Expand Up @@ -290,6 +325,13 @@ export const LOCAL_APPS = {
displayOnModelPage: isLlamaCppGgufModel,
snippet: snippetNodeLlamaCppCli,
},
llamafile: {
prettyLabel: "llamafile",
docsUrl: "https://github.com/Mozilla-Ocho/llamafile",
mainTask: "text-generation",
displayOnModelPage: isLlamaFileModel,
snippet: snippetLlamafile,
},
vllm: {
prettyLabel: "vLLM",
docsUrl: "https://docs.vllm.ai",
Expand Down
Loading