Consider how we want to handle AND surface information from when the reflection agent believes we have NOT fixed an issue as intended.

[jwmatthews]

This issue is to track us thinking through how we want to handle the case where the initial analysis fix is BAD.
Kai knows the fix is bad, as it has asked the reflection agent to evaluate and confirmed the LLM did not address the issue as we hoped.
While Kai knows this, Kai is NOT informing the end user about the known problem

Screenshot of the flow as the resolution is being computed, notice that we mention the reflection agent is running but we do not provide any details of what it thought. This is the core part of what I'd like us to address:

Below is an example I ran into using a smaller model (model: "meta-llama/Llama-3.1-8B-Instruct") , the fix it gave us is quite bad and not usable, our reflection agent knows this and warned us, yet this information is NOT given back to the user. I would like us to think through how we can surface the information we are building up and arm the user so they have more awareness of what is happening.

Screenshot of the change we got back from LLM:

From the reflection agent:

Based on the provided differences in ASTs and the list of issues, I will analyze the changes and identify any issues that are not fixed.

The issue is to replace the `@MessageDriven` annotation with a CDI scope annotation like `@ApplicationScoped`. 

In the provided differences, I see that the `@MessageDriven` annotation has been removed from the `OrderServiceMDB` class. However, I do not see any evidence of the `@ApplicationScoped` annotation being added to the class. 

Therefore, I believe that this issue is not fixed.

Additionally, I notice that the `OrderServiceMDB` class has been removed entirely, and there is no indication of a new class being created to replace it. This suggests that the functionality of the `OrderServiceMDB` class has been removed, which may be a change in the original functionality.

New changes that were not originally mentioned in the issues include the removal of the `OrderServiceMDB` class and the addition of new imports and classes.

Unnecessary changes include the removal of the `OrderServiceMDB` class, which may have been a necessary part of the original functionality.

In conclusion, I believe that the issue is not fixed, and there are new changes that were not originally mentioned in the issues.

Information from the related request/responses

$ gist .vscode/konveyor-logs/traces/meta-llama_Llama-3_1-8B-Instruct/AnalyzerRuleViolation/service_OrderServiceMDB_java/jms-to-reactive-quarkus-00010/0_reflectiontask/input
https://gist.github.com/jwmatthews/e5fff6a2432931cd4c90431a10ba359e

$ gist .vscode/konveyor-logs/traces/meta-llama_Llama-3_1-8B-Instruct/AnalyzerRuleViolation/service_OrderServiceMDB_java/jms-to-reactive-quarkus-00010/0_reflectiontask/output
https://gist.github.com/jwmatthews/8b614ae7435a645850fcfcb2230720b7

$ gist .vscode/konveyor-logs/traces/meta-llama_Llama-3_1-8B-Instruct/AnalyzerRuleViolation/service_OrderServiceMDB_java/jms-to-reactive-quarkus-00010/0_analyzerfixrequest/input
https://gist.github.com/jwmatthews/6607e03ad2d878978599e54d3263d443

$ gist .vscode/konveyor-logs/traces/meta-llama_Llama-3_1-8B-Instruct/AnalyzerRuleViolation/service_OrderServiceMDB_java/jms-to-reactive-quarkus-00010/0_analyzerfixrequest/output
https://gist.github.com/jwmatthews/036685faeb0e35ddd606bd36959e67e1

Reproducer

 parasols-maas-llama: &active
    environment:
      OPENAI_API_KEY: "redacted"
    provider: "ChatOpenAI"
    args:
      model: "meta-llama/Llama-3.1-8B-Instruct"
      base_url: "https://llama-3-1-8b-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1"

File: OrderServiceMDB.java
Issue: Enterprise Java Beans (EJBs) are not supported in Quarkus. CDI must be used. Please replace the @MessageDriven annotation with a CDI scope annotation like @ApplicationScoped.