feat(ai): Enable Context Variables for Chat Requests

[planger]

What it does

This PR introduces the ability for users to add contextual information to chat requests. This allows manually passing elements such as files, symbols, or other variables, which chat agents can then utilize when generating responses.

Context Variables

To represent contextual elements, we extend the existing variables concept. Previously, variables were placeholders within chat request text (e.g., #selectedText), dynamically resolved by the chat service before processing (ResolvedAIVariable.value), replacing the occurrence of the variable directly in the user input.

With this PR, we introduce context variables (AIContextVariable), which enhance the existing variables by not only resolving a value for text replacement but also attaching a dedicated context value (ResolvedAIContextVariable.contextValue) to the chat request. This enables agents to use contextual information without making it a fixed part of the user’s input.

Key Benefits:

• Separation of concerns: Large context elements can be included without entangling them in the user’s message.
• Flexible processing strategies: Agents can use different techniques, such as summarizing or fetching context on demand.
• Persistent context: The context can remain stable across multiple requests within a conversation without having to re-add them.

Context Variable Resolution

Context variable requests (AIVariableResolutionRequest) can be passed alongside a ChatRequest to the ChatService. The chat service then resolves context variables and attaches them to the ChatRequestModel under the context property.

If a context variable is referenced in the chat request (e.g., #file:abc.txt), it serves a dual purpose:

1. Text replacement: ResolvedAIVariable.value replaces the variable in the chat text.
2. Context attachment: ResolvedAIContextVariable.contextValue is added to the chat request model.

Attaching Context Variables

Context variables can be added to a chat request in two ways:

1. Explicitly: By providing a list of AIVariableResolutionRequest objects to ChatService.sendRequest alongside the ChatRequest.
2. Implicitly: By mentioning a context variable in the chat text (e.g., #file:abc.txt). The chat service automatically parses it into a variable request and attaches it to the context.

User Interface Enhancements

To support context variables, the chat input widget now maintains a list of added context variables. This list appears below the text input and utilizes Theia’s label provider mechanism for user-friendly display (including labels, additional information, and icons). Users can add context variables through:

1. Quick Pick Dialog (+ button): Opens a selection dialog listing all registered context variables. To improve usability, the AIVariable interface now includes optional label and icon properties.
2. Inline Input (# prefix): Users can type context variables directly into the chat input. The existing code completion mechanism suggests context variables, and selecting one also adds it to the context variable list.

Variable Arguments

Variables support arguments (parameters) that influence their resolution. This PR enhances argument handling by introducing:

• AIVariableArgPicker: Invoked when users add context variables via the Quick Pick dialog (+ button).
• AIVariableArgCompletionProvider: Invoked when users type context variables manually.

With these two mechanisms, variable providers can choose how to support users in selecting the right arguments via a UI or via auto-completion.

Example: File Context Variable (#file)

To demonstrate these improvements, we introduce a new context variable #file, which accepts a file path as an argument. We also register:

• An argument picker for selecting files when adding #file via the Quick Pick dialog.
• A completion provider that allows users to type # and get file path suggestions directly.

Therefore, we've refactored existing code into a QuickFileSelectService to provide a consistent file picker experience for both argument pickers and completion providers as well as for the existing file search (Ctrl-p).

Utilizing Chat Request Context

Chat agents can determine how they process the provided context. Common approaches include:

1. Summarization: The agent summarizes the manually specified context (e.g., compiling a list) before passing it to the LLM.
2. Context Window Management: The agent dynamically decides how much context to include:
   • If the entire context fits within the model’s context limit, it is included as-is.
   • If the context is too large, the agent applies ranking and summarization mechanisms to extract the most relevant parts.
   • Advanced implementations may use a multi-turn prompt flow to refine which elements are most relevant before sending them to the LLM.
3. On-Demand Retrieval: Instead of including the full context upfront, the agent exposes tool functions to the LLM, allowing it to retrieve specific context elements dynamically as needed. This PR includes tool functions to do that.

Fixes #14839

How to test

Use the + icon to add context files

adding-context-plus.webm

Use the context variable auto complete

adding-context-auto-complete.webm

Use the context variable argument auto complete

adding-context-auto-complete-arg.webm

Use drag and drop to add files to the context

adding-context-drop.webm

Follow-ups

• [Theia AI] Integrate context feature with Coder #14895
• [Theia AI] Resolve file context variable for Universal and Workspace #14945

Breaking changes

• This PR introduces breaking changes and requires careful review. If yes, the breaking changes section in the changelog has been updated.

Attribution

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[sdirix]

Thanks for the work. I had a first look.

I tested on aea0e4d with the fix from #14942 cherry-picked on top

Sadly I have some usability issues:

Too narrow chat view breaks variable

Text and Variable

Adding variableText, e.g. file:my-path AND a marker to the chat input is unintuitive. It's not clear to me whether I need to keep the text or whether I can delete it and what the difference is when keeping the text and not. Also sending a message will get rid of the text in the prompt either way for the follow up message, however the context is still rendered.

Not all agents get the content

I would have expected that no matter what agent is used, they would get the content of the file injected in some way. However Universal does not seem to have access / care about the variables

Drag and Drop uses a different variable

Drag and Drop adds files via file-provider:path instead of file:path

Switching threads

Switching sessions or creating a new one does not clear the context

[JonasHelming]

the two follow ups for this: #14895 and #14945
(To integrate with coder, universal and workspace)
are tracked!

[planger]

@sdirix Thank you very much for the review! I've addressed most of your comments (see my direct inline replies).

Too narrow chat view breaks variable

Oh, right, thanks. I've fixed this in a8c2966.

Text and Variable

There are two separate things:

1. The user input (which may or may not use variables)
2. The context provided to the agent (which may or may not contain context variables)

Variables in the input get resolved to a value that will replace the occurrence of the variable in the request text.

Context variables are resolved to a contextValue and are additional information available for the agent to incorporate in whatever way they decide in their flow. Whether it is they just announce this as a list initially in their system message to the LLM and allow it to resolve this later on via tool functions, whether they summarize it upfront, or include it verbatim as they decide it is small enough for their active context management --- that all is up to the agent. And this is really where context variables add value over plain variables, which always need to be directly replaced in the user input, because this a more structured way of adding large data, still enabling to refer to the data in the user prompt but keep them separated and not intermingled with the direct user request.

Edit: A variable can be just a plain variable (it doesn't provide a contextValue or it can be both, a normal variable and a context variable. It cannot be a context variable alone, as this simplifies referring to the context variable in the user input. It is expected that the context variable's value then resolves to some sort of identifier of the context (e.g. workspace relative file URI).

If now a user drags a context variable into the text input, it has two effects: it'll add the variable in the user input (to be replaced by variable.value) AND add it to the context (with its variable.contextValue). If the user drags it into the chat input widget but not in the text, it'll only add it into the context.

I think this distinction is very valuable, because it allows users to add elements into the context and mention them in the text, while the framework has the flexibility to allow replacing the occurrence in the text input with a different (shorter) value, such as a name or identifier, than in the context.

At the example of how a file agent could use this:
A user can add several files in to the context, e.g. the files they currently work on, or files that contain relevant typings. In addition the user can refer to the file in the text input, e.g. fix this or that in file #file:a.txt. The occurrence of the variable in the text will be replaced into the identifier that is common between the context elements and the chat request (the workspace relative path for files). The agent sends this message along with a list of the context elements (e.g. their identifier and a summary of the contents) and offers tool functions for the LLM to obtain the full content.

Of course, I see your point that it is not quite obvious, as both seem equivalent and are synchronously added when I type something, but then not removed when the corresponding text in the input is removed. Not sure how to best improve the situation though. Introducing a flag that remembers which context was added automatically and then remove this too, when we remove the text? But this felt a bit over-complicated.

Not all agents get the content

Right, but coder will use it (#14895). (Comment from Jonas: Also universal and workspace: #14945)
Whether other Theia IDE agents use it or not is still up for debate. We may want to add prompt fragments and tool functions that make it very easy to use it by default. This is also why I added the tool functions already to make it easy to extend existing agents with context capabilities.

Drag and Drop uses a different variable

Right, I think this was caused by a parallel change. I've fixed this now for consistency.

Switching threads

Right, my feeling was this is a good thing. It'd be easy to change, but we also don't clear the text input when switching threads, so I felt is is more consistent this way and opens up further use cases (e.g. starting a new thread with the list of files that I currently work on). But I'm open to what others think!

Thanks again for your review!

[planger]

@sdirix Sorry for the force-push. I had to rebase and resolve conflicts after #14716.

In addition (6f72322), I made the code completion now also trigger at : as this is a sensible way of filtering down to the type of variable first and the obtain completions. Therefore, I also made the completion providers more powerful, as they can take over the entire completion without having the chat language contribution prefix the insertion text with the variable name.

[sdirix]

Thanks for the detailed explanations. Following them I was able to leverage the context

As far as I understand we will add the ability to understand the context to all agents we offer by default. This is then fine with me as a first implementation.

Some follow ups to consider to improve the UX:

• All agents will always run the tool function to discover the list of context items, no matter whether something is placed there. This pollutes the chat. Maybe there should be a mechanism to dynamically add the context tool functions only if there is actually a context in the request.
• In case an agent has a LLM without tool support configured, we should highlight that the context will not be used, e.g. with a grey transparent overlay
• One step further: If we have a prompt without the context tool functions and without that dynamic behavior outlined above, we should also disable / mark the context as not in use.

I think it's very important for the release that we have a clear and understandable UX for the context as otherwise we will get a lot of issues reported that the feature is not working.

[JonasHelming]

Thanks for the detailed explanations. Following them I was able to leverage the context

As far as I understand we will add the ability to understand the context to all agents we offer by default. This is then fine with me as a first implementation.

Some follow ups to consider to improve the UX:

• All agents will always run the tool function to discover the list of context items, no matter whether something is placed there. This pollutes the chat. Maybe there should be a mechanism to dynamically add the context tool functions only if there is actually a context in the request.

I am not sure I get this. In Workspace and Coder, they will not have to run such as function, as we place the context list in a variable in the prompt, there will be only a tool call if the LLM looks at content.

• In case an agent has a LLM without tool support configured, we should highlight that the context will not be used, e.g. with a grey transparent overlay

=> Very valid case, for universal, it will work, as we resolve the context. Coder and Workspace won't actually work at all without functions.

• One step further: If we have a prompt without the context tool functions and without that dynamic behavior outlined above, we should also disable / mark the context as not in use.

=> Yes, it is a bit tricky, I will track this

I think it's very important for the release that we have a clear and understandable UX for the context as otherwise we will get a lot of issues reported that the feature is not working.

[planger]

@sdirix Thank you very much for your review and for adding the translation!

I think it's very important for the release that we have a clear and understandable UX for the context as otherwise we will get a lot of issues reported that the feature is not working.

I absolutely agree. Doing this upfront, however, may require a lot of flags that each agent needs to provide. I think one option that is easier to do generically is to add an indicator in the response which context was processed.

We could track the variable resolution for a request in the chat service and the tool functions could notify the chat model that they've consumed a specific context. We could add an additional processedContext property in the chat response model that is written to by the chat service on variable resolution and by the tool functions and show the tracked context accesses in the chat response.

WDYT?

[JonasHelming]

@sdirix Thank you very much for your review and for adding the translation!

I think it's very important for the release that we have a clear and understandable UX for the context as otherwise we will get a lot of issues reported that the feature is not working.

I absolutely agree. Doing this upfront, however, may require a lot of flags that each agent needs to provide. I think one option that is easier to do generically is to add an indicator in the response which context was processed.

We could track the variable resolution for a request in the chat service and the tool functions could notify the chat model that they've consumed a specific context. We could add an additional processedContext property in the chat response model that is written to by the chat service on variable resolution and by the tool functions and show the tracked context accesses in the chat response.

WDYT?

very interesting idea! if we do this, I believe the notification should happen in the context itself, as we can also consume the context via variables or API, not only via functions.

[sdirix]

I am not sure I get this. In Workspace and Coder, they will not have to run such as function, as we place the context list in a variable in the prompt, there will be only a tool call if the LLM looks at content.

I see, I was using the tool function to look a the content which is part of this PR. Using a variable is better 👍

[planger]

very interesting idea! if we do this, I believe the notification should happen in the context itself, as we can also consume the context via variables or API, not only via functions.

Exactly, I think we need the API on the mutable chat response and then have the chat service (on variable resolution) and the tool functions (on invocation) use this API to report that context has been resolved.

[sdirix]

very interesting idea! if we do this, I believe the notification should happen in the context itself, as we can also consume the context via variables or API, not only via functions.

Exactly, I think we need the API on the mutable chat response and then have the chat service (on variable resolution) and the tool functions (on invocation) use this API to report that context has been resolved.

I like the idea!

Each response could have a copy of the context including the marker which of it was used. Then we could show for each response what was in the context and what was used, for example in a collapsible section, in a rich hover over etc.

[planger]

I've extracted #14961

[colin-grant-work]

@planger With this change, it looks like this file should be moved to the browser folder, as its types now require DOM typing, and it couldn't reasonably be instantiated on the backend.

[planger]

Thanks for raising this issue! You are right, this is an oversight.
I think it'd be best if we could avoid moving the variable service entirely to the frontend. How about adding an additional contribution point with a frontend only binding, and have the variable service on the frontend collect the contributions in addition to what the common variable service does?

[planger]

I've suggested a fix here: #14969 Thanks again for raising this issue!