[Bug] Parsing of 'updated file' contents looks broken for some situations - unable to handle multiple ways of expressing 'Updated File'

[jwmatthews]

Using model: mistralai/mixtral-8x7b-instruct-v01 from IBM BAM

I am running example/run_demo.py

run_demo.py attempts to migrate 26 files, of this I see 6 failures, as in after exhausting all retries I see 6 files with messages of:
[pom.xml] Received status code 500

These are the 6 files where Kai is saying the LLM did not return an updated file,

• src/main/java/com/redhat/coolstore/model/InventoryEntity.java
• src/main/java/com/redhat/coolstore/model/ShoppingCart.java
• src/main/java/com/redhat/coolstore/service/PromoService.java
• src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java
• src/main/java/com/redhat/coolstore/rest/CartEndpoint.java
• pom.xml

This is reported as:

 "The LLM did not provide an updated file for X"
 "No codeblocks detected in LLM response"

yet as I look at the debug output I can see the LLM did return an updated file.

Example of LLM Result info for CartEndPoint.java
Example of LLM Result info for pom.xml

To aid debugging I captured the full console output in 2 separate gists below:

• LOGLEVEL=debug make run-server - Full output in gist
• ./run_demo.py - Full output in gist

Example of kai/config.toml

[models]
provider = "ChatIBMGenAI"

[models.args]
model_id = "mistralai/mixtral-8x7b-instruct-v01"

[jwmatthews]

I downloaded the full console output from "LOGLEVEL=debug make run-server" and then searched for the unique ways 'Updated File' is mentioned:

https://gist.githubusercontent.com/jwmatthews/c0c94b5635200ccfdd1f7595cfaecdae/raw/7c5425121f3b91994037fa64d6d7c6f6df0e4b91/2024_June-24_kai_console_output.txt

$ cat ./2024_June-24_kai_console_output.txt | grep "Updated" | sort -u
# Updated File
## Updated File
**Updated File**

I assume the root issue is that https://github.com/konveyor-ecosystem/kai/blob/main/kai/models/file_solution.py assume we will consistently see the Updated File represented in a section of "## Updated File"

[pranavgaikwad]

I have seen this too with Mixtral very often. As a workaround, I added following to the prompt:

// rest of the original prompt

{% if model_provider.model_id == 'mistralai/mixtral-8x7b-instruct-v01' %}
Only use ## to begin section headers in the markdown above.
{% endif %}

// llama header line

[fabianvf]

This should be fixed, please reopen if you're still seeing it.