What is the difference between chat-instruct mode and instruct mode?

[majestichou]

In instruct and chat-instruct modes, the instruction template selected under Parameters > Instruction template must match the current model. What is the difference between chat-instruct mode and instruct mode?

[TechScorpion]

As far as I know, chat-instruct and instruct modes have different templates for how they will respond (prompt templates?). And I think chat-instruct still takes stuff from the character profile (which the chat mode does too) but I don't think the instruct mode does. You could check though (see if there's a difference in context size when asking a question or asking it to write something in a new chat). So instruct mode is more general (what they write wouldn't be from a particular character's point of view).

[araleza]

There are two types of models available, base models and instruct models. Base models are trained to continue what's been written so far, whereas instruct models are trained to carry out the instruction / answer the question.

Base models might seem like a more natural fit for chatting with, especially if they've been trained on a roleplay dataset, as they can see a conversation is taking place and continuing to make conversation is what they'll tend to do automatically. Instruct models will not tend to understand what to do, because they will be looking for an instruction or question, and that's not what chats looks like.

Chat-instruct mode is used when you want to chat with an instruct trained model. Behind the scenes, the instruct model is given an instruction to 'write the next reply in the chat', which is an instruction it can follow to effectively be able to carry out a conversation. You can see that if you do a chat-instruct mode conversation, and then choose 'Send to default' from that little menu box in the corner.

So this Chat-Instruct conversation (with Mixtral-8x7B-instruct):

has the instruction:

   'Continue the chat dialogue below. Write a single reply for the character "Chiharu Yamada".'

inserted at the top (seen here after 'Send to default' is used), before the character card (which includes an example conversation), and before the actual current conversation:

If we did the same with Chat mode rather than Chat-Instruct, you can see this in the Default tab instead:

Notice that the conversation still worked in chat mode, even though this is an instruct variant of Mixtral, and no instruction was given. Also, the Mixtral instruct model is looking for [INST] and [/INST] tags around its inputs, and they're just not there in chat mode. (Other instruct models may look for e.g. '### Instruction:', '### Input:', and '### Response:' markers, known as Alpaca format). But despite the [INST] and [/INST] tags being missing, it all still happens to work in this case. That's likely just because models tend to be pretty flexible, as long as there's enough to work out what they should be doing.

One final point to note is that chat-instruct with instruct models can sometimes get you longer replies. That's because base models have learned from datasets where people are chatting, and it's rare for people to type more than a sentence or two when chatting. But instruct models have been fine tuned on datasets where they get a short question, and write multiple paragraphs in reply. So by using an instruct model, you can often get more lengthy replies, and Chat-Instruct mode can help the model lean in that direction further.

So whether you use a base model in Chat mode or an instruct model in Chat-Instruct mode depends on your personal preference, how the particular model you're using has been trained (i.e. whether it has many chat conversations of the type you like in its training dataset), and how lengthy you want its replies to be.

And just to answer your question, here's Instruct (not Chat-Instruct) mode:

which leads to this being passed to the model (and including the model's output):

@TechScorpion is correct that the prompt template is being used (so Mixtral's [INST] and [/INST] tags are present), and the character profile is not present.

[majestichou]

Thanks a lot !! Thank you for your detailed answer. I understand.