-
Notifications
You must be signed in to change notification settings - Fork 144
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- [Docs]: Added a separate examples page (WIP)
- [Docs]: Added the `LLM as Chatbot` example
- Loading branch information
1 parent
383c1ae
commit 416212b
Showing
10 changed files
with
233 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| template: examples.html | ||
| title: Examples | ||
| hide: | ||
| - navigation | ||
| - toc | ||
| - footer | ||
| --- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| # LLM as Chatbot | ||
|
|
||
| { width=800 } | ||
|
|
||
| This [example](https://github.com/deep-diver/LLM-As-Chatbot) is built by Chansung Park. It can run any open-source LLM either as a Gradio chat app or as a Discord bot. | ||
| With `dstack`, you can run this Gradio chat app or Discord bot in any cloud with a single command. | ||
| To give it a try, follow the instructions below. | ||
|
|
||
| ## 1. Define a profile | ||
|
|
||
| ??? info "Prerequisites" | ||
| Before running the example, ensure that you have [installed](../docs/installation/pip.md) `dstack` and [configured a project](../docs/guides/projects.md) | ||
| to use your preferred cloud account (AWS, GCP, Azure, or Lambda Cloud). | ||
|
|
||
| Each LLM model requires specific resources. To inform dstack about the required resources, you need to define a profile | ||
| in the `.dstack/profiles.yaml` file within your project. | ||
|
|
||
| Each profile must include the project name, and you have the option to specify the GPU name, its memory, instance type, | ||
| retry policy, and more. Check the [reference](../docs/reference/profiles.yml.md) for more details. | ||
|
|
||
| <div editor-title=".dstack/profiles.yml"> | ||
|
|
||
| ```yaml | ||
| profiles: | ||
| - name: gcp-t4 | ||
| project: gcp | ||
| resources: | ||
| gpu: | ||
| name: T4 | ||
| default: true | ||
| ``` | ||
| </div> | ||
| If you use this profile, dstack will utilize the project named `gcp` and a cloud instance that has an NVIDIA T4 GPU. | ||
|
|
||
| ## 2. Run a Gradio app | ||
|
|
||
| Here's the configuration that runs the Gradio app: | ||
|
|
||
| <div editor-title="gradio.dstack.yml"> | ||
|
|
||
| ```yaml | ||
| type: task | ||
| env: | ||
| # (Optional) Specify your Hugging Face token | ||
| - HUGGING_FACE_HUB_TOKEN= | ||
| # (Optional) Specify your Serper API Key | ||
| - LLMCHAT_SERPER_API_KEY= | ||
| ports: | ||
| - 6006 | ||
| commands: | ||
| - pip install -r requirements.txt | ||
| - LLMCHAT_APP_MODE=GRADIO python entry_point.py | ||
| ``` | ||
|
|
||
| </div> | ||
|
|
||
| Here's how you run it with `dstack`: | ||
|
|
||
| <div class="termy"> | ||
|
|
||
| ```shell | ||
| $ dstack run . -f gradio.dstack.yml | ||
| dstack will execute the following plan: | ||
| CONFIGURATION PROJECT INSTANCE RESOURCES SPOT | ||
| gradio.dstack.yml gcp n1-highmem-2 2xCPUs, 13312MB, 1xT4 auto | ||
| Continue? [y/n]: y | ||
| Provisioning and establishing an SSH tunnel... | ||
| Running on local URL: http://127.0.0.1:6006 | ||
| To interrupt, press Ctrl+C... | ||
| ``` | ||
|
|
||
| </div> | ||
|
|
||
| After you confirm, `dstack` will provision the cloud instance, run the task, and forward the defined ports to your local | ||
| machine for secure and convenient access. | ||
|
|
||
| { width=800 } | ||
|
|
||
| !!! info "NOTE:" | ||
| To use a non-default profile, simply specify its name with `--profile NAME` when using `dstack run`. | ||
|
|
||
| ## 3. Run a Discord bot | ||
|
|
||
| Here's the configuration that runs the Gradio app: | ||
|
|
||
| <div editor-title="discord.dstack.yml"> | ||
|
|
||
| ```yaml | ||
| type: task | ||
| env: | ||
| # (Required) Specify your Discord bot token. | ||
| - DISCORD_BOT_TOKEN= | ||
| # (Required) Specify the name of the model. See `README.md`` for supported models. | ||
| - DISCORD_BOT_MODEL_NAME=alpaca-lora-7b | ||
| # (Optional) Specify your Hugging Face token | ||
| - HUGGING_FACE_HUB_TOKEN= | ||
| # (Optional) Specify your Serper API Key to enable Internet search support. | ||
| - LLMCHAT_SERPER_API_KEY= | ||
|
|
||
| commands: | ||
| - pip install -r requirements.txt --progress-bar off | ||
| - LLMCHAT_APP_MODE=DISCORD python entry_point.py | ||
| ``` | ||
| </div> | ||
| ??? info "How to acquire a Discord bot token" | ||
| Before running, ensure you have specified your Discord bot token, which you can obtain from the [Discord Developer | ||
| Portal](https://discord.com/developers/docs/intro). If you haven't set up a Discord Bot on the portal yet, | ||
| follow the [How to Create a Discord Bot Account](https://www.freecodecamp.org/news/create-a-discord-bot-with-python/) | ||
| section of the tutorial from freeCodeCamp. | ||
| Finally, here's how you run it with `dstack`: | ||
|
|
||
| <div class="termy"> | ||
|
|
||
| ```shell | ||
| $ dstack run . -f discord.dstack.yml | ||
| dstack will execute the following plan: | ||
| CONFIGURATION PROJECT INSTANCE RESOURCES SPOT | ||
| discord.dstack.yml gcp n1-highmem-2 2xCPUs, 13312MB, 1xT4 auto | ||
| Continue? [y/n]: y | ||
| Provisioning... | ||
| To interrupt, press Ctrl+C... | ||
| ``` | ||
|
|
||
| </div> | ||
|
|
||
| Once you confirm, `dstack` will provision the cloud instance and run the task. Once it's up, you can freely send messages | ||
| to your bot via Discord. | ||
|
|
||
| { width=800 } | ||
|
|
||
| For advanced commands supported by the bot, check the [README](https://github.com/deep-diver/LLM-As-Chatbot#discord-bot) file. | ||
|
|
||
| [Source code](https://github.com/deep-diver/LLM-As-Chatbot){ .md-button .md-button--github } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| {% extends "main.html" %} | ||
|
|
||
| {% block content %} | ||
| <section class="tx-container"> | ||
| <div class="md-grid md-typeset"> | ||
| <div class="-landing__highlights"> | ||
| <div class="tx-landing__highlights_text"> | ||
| <h2>Examples</h2> | ||
| </div> | ||
|
|
||
| <div class="tx-landing__highlights_grid"> | ||
| <a href="llmchat"> | ||
| <div class="feature-cell"> | ||
| <div class="feature-icon"> | ||
| <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"> | ||
| <path d="M12 3c5.5 0 10 3.58 10 8s-4.5 8-10 8c-1.24 0-2.43-.18-3.53-.5C5.55 21 2 21 2 21c2.33-2.33 2.7-3.9 2.75-4.5C3.05 15.07 2 13.13 2 11c0-4.42 4.5-8 10-8m5 9v-2h-2v2h2m-4 0v-2h-2v2h2m-4 0v-2H7v2h2Z"></path> | ||
| </svg> | ||
| </div> | ||
| <h3> | ||
| LLM as Chatbot | ||
| </h3> | ||
|
|
||
| <p> | ||
| Run an open-source LLM of your choice either as a Gradio chat app or as a Discord bot. | ||
| </p> | ||
| </div> | ||
| </a> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </section> | ||
| <br> | ||
| <br> | ||
| <br> | ||
| <br> | ||
| {% endblock %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters