Skip to content

Commit

Permalink
📝 Recommend starcoder 2 7b
Browse files Browse the repository at this point in the history
  • Loading branch information
sestinj committed Apr 7, 2024
1 parent ebb9650 commit 889e552
Showing 1 changed file with 22 additions and 6 deletions.
28 changes: 22 additions & 6 deletions docs/docs/walkthroughs/tab-autocomplete.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,28 @@

Continue now provides support for tab autocomplete in [VS Code](https://marketplace.visualstudio.com/items?itemName=Continue.continue) and [JetBrains IDEs](https://plugins.jetbrains.com/plugin/22707-continue/edit). We will be greatly improving the experience over the next few releases, and it is always helpful to hear feedback. If you have any problems or suggestions, please let us know in our [Discord](https://discord.gg/vapESyrFmJ).

## Setting up with Starcoder 2 7b (recommended)

If you want to have the best autocomplete experience, we recommend using Starcoder 2 7b, which is available through [Fireworks AI](https://fireworks.ai/models/fireworks/starcoder-7b). To do this, obtain an API key and add it to your `config.json`:

```json
{
"tabAutocompleteModel": {
"title": "Starcoder 2",
"provider": "openai",
"model": "accounts/fireworks/models/starcoder-7b",
"apiBase": "https://api.fireworks.ai/inference/v1",
"apiKey": "YOUR_API_KEY"
}
}
```

## Setting up with Ollama (default)

We recommend setting up tab-autocomplete with a local Ollama instance. To do this, first download the latest version of Ollama from [here](https://ollama.ai). Then, run the following command to download our recommended model:

```bash
ollama run starcoder:3b
ollama run starcoder2:3b
```

Once it has been downloaded, you should begin to see completions in VS Code.
Expand All @@ -21,7 +37,7 @@ All of the configuration options available for chat models are available to use
"tabAutocompleteModel": {
"title": "Tab Autocomplete Model",
"provider": "ollama",
"model": "starcoder:3b",
"model": "starcoder2:3b",
"apiBase": "https://<my endpoint>"
},
...
Expand All @@ -32,7 +48,7 @@ If you aren't yet familiar with the available options, you can learn more in our

### What model should I use?

If you are running the model locally, we recommend `starcoder:3b`.
If you are running the model locally, we recommend `starcoder2:3b`.

If you find it to be too slow, you should try `deepseek-coder:1.3b-base`.

Expand All @@ -46,7 +62,7 @@ The following can be configured in `config.json`:

### `tabAutocompleteModel`

This is just another object like the ones in the `"models"` array of `config.json`. You can choose and configure any model you would like, but we strongly suggest using a small model made for tab-autocomplete, such as `deepseek-1b`, `starcoder-1b`, or `starcoder-3b`.
This is just another object like the ones in the `"models"` array of `config.json`. You can choose and configure any model you would like, but we strongly suggest using a small model made for tab-autocomplete, such as `deepseek-1b`, `starcoder-1b`, or `starcoder2-3b`.

### `tabAutocompleteOptions`

Expand All @@ -70,7 +86,7 @@ This object allows you to customize the behavior of tab-autocomplete. The availa
"tabAutocompleteModel": {
"title": "Tab Autocomplete Model",
"provider": "ollama",
"model": "starcoder:3b",
"model": "starcoder2:3b",
"apiBase": "https://<my endpoint>"
},
"tabAutocompleteOptions": {
Expand All @@ -93,7 +109,7 @@ Follow these steps to ensure that everything is set up correctly:

1. Make sure you have the "Enable Tab Autocomplete" setting checked (in VS Code, you can toggle by clicking the "Continue" button in the status bar).
2. Make sure you have downloaded Ollama.
3. Run `ollama run starcoder:3b` to verify that the model is downloaded.
3. Run `ollama run starcoder2:3b` to verify that the model is downloaded.
4. Make sure that any other completion providers are disabled (e.g. Copilot), as they may interfere.
5. Make sure that you aren't also using another Ollama model for chat. This will cause Ollama to constantly load and unload the models from memory, resulting in slow responses (or none at all) for both.
6. Check the output of the logs to find any potential errors (cmd/ctrl+shift+p -> "Toggle Developer Tools" -> "Console" tab in VS Code, ~/.continue/core.log in JetBrains).
Expand Down

0 comments on commit 889e552

Please sign in to comment.