Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error parsing updated file when LLM instructs us to remove the file. #218

Open
jwmatthews opened this issue Jun 29, 2024 · 2 comments
Open
Labels
agent enhancement New feature or request prompt-engineering Issues related to changes for prompts or tweaks for specific models
Milestone

Comments

@jwmatthews
Copy link
Member

Model: llama-3-70b-instruct

Below is an example when we are not able to understand the LLM's instructions for removing a file
The LLM is advising us that:
The "web.xml" file is not needed in Quarkus, so we can remove it. Instead, we can use annotations to configure the application.

We generate this Exception:

WARNING - 2024-06-29 10:04:22,425 - [   llm_io_handler.py:260  - get_incident_solutions_for_file()] - Request to model failed for batch 1/1 for src/main/webapp/WEB-INF/web.xml with exception, retrying in 10s
Error in LLM Response: The LLM did not provide an updated file for src/main/webapp/WEB-INF/web.xml
DEBUG - 2024-06-29 10:04:22,426 - [   llm_io_handler.py:263  - get_incident_solutions_for_file()] - Traceback (most recent call last):
  File "/Users/jmatthews/git/jwmatthews/kai/kai/llm_io_handler.py", line 254, in get_incident_solutions_for_file
    raise Exception(
Exception: Error in LLM Response: The LLM did not provide an updated file for src/main/webapp/WEB-INF/web.xml

Example of LLM result (I trimmed a ' from the XML snippet section for rendering in github issue)

## Reasoning

The issue identified is related to session replication, which is not suitable for a cloud environment. To address this, we need to disable HTTP session clustering. In Quarkus, we can achieve this by using the `@SessionScoped` annotation from the `jakarta.enterprise.context` package instead of relying on the `distributable` element in the `web.xml` file.

Since we are disabling session replication, we don't need the `distributable` element in the `web.xml` file. We can remove this element from the file.

Additionally, we need to update the `pom.xml` file to include the necessary dependencies for Quarkus. We need to add the `jakarta.enterprise.context` dependency to the `pom.xml` file.

## Updated File

The `web.xml` file is not needed in Quarkus, so we can remove it. Instead, we can use annotations to configure the application.

## Additional Information

In the `pom.xml` file, we need to add the following dependency:
``xml
<dependency>
    <groupId>jakarta.enterprise</groupId>
    <artifactId>jakarta.enterprise.context-api</artifactId>
</dependency>
``
We also need to update the imports in the Java files to use the `jakarta.enterprise.context` package instead of the `javax.enterprise.context` package.

Note: We are not providing an updated Java file in this step, as the issue is related to the `web.xml` file. In subsequent steps, we will update the Java files accordingly.

Below are links to the full Prompt and the full LLM result
$ gist prompt
https://gist.github.com/jwmatthews/40217e662a79ee6597897a150357a257

$ gist llm_result
https://gist.github.com/jwmatthews/725dd3ae2df444e2781640127831d363

@jwmatthews jwmatthews added the bug Something isn't working label Jun 29, 2024
@shawn-hurley shawn-hurley added enhancement New feature or request agent prompt-engineering Issues related to changes for prompts or tweaks for specific models and removed bug Something isn't working labels Jan 10, 2025
@shawn-hurley
Copy link
Contributor

I am moving this to an enhancement because it will require an agent and prompt engineering work to tell the IDE what it can do.

@fabianvf
Copy link
Contributor

We should attempt to parse the codeblock only out of the Updated File section in addition to the Additional Information agent @shawn-hurley mentioned

@fabianvf fabianvf added this to the v0.2.0 milestone Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent enhancement New feature or request prompt-engineering Issues related to changes for prompts or tweaks for specific models
Projects
None yet
Development

No branches or pull requests

3 participants