Partial response with mixtral: Updated file is incomplete.

[jwmatthews]

Below is an example where we parsed the Updated File out of the LLM results, yet the contents of the file look incomplete, see how this stops early.

Using

provider = "IBMOpenSource"
args = { model_id = "ibm-mistralai/mixtral-8x7b-instruct-v01-q" }

   public static Order jsonToOrder(String json) {
        JsonReader jsonReader = Json.createReader(new StringReader(json));
        JsonObject rootObject = jsonReader.readObject();
        Order order = new Order();
        order.setCustomerName(rootObject.getString("customerName"));
        order.setCustomerEmail(rootObject.getString("customerEmail"));
        order.setOrderValue(rootObject.getJsonNumber("orderValue").doubleValue());
        order.setRetailPrice(rootObject.getJsonNumber("retailPrice").doubleValue());
        order.setDiscount(rootObject.getJsonNumber("dis

$ gist Transformers.java
https://gist.github.com/jwmatthews/6734acbdeef7751b561afd4ab466a805

$ gist Transformers.java.llm_result.md
https://gist.github.com/jwmatthews/067c5fb8af58d738dff432d26cfca81b

$ gist Transformers.java.prompts.md
https://gist.github.com/jwmatthews/ef35e0e4d93385b0d5021f119e542974

[JonahSussman]

I'm wondering if we ran out of tokens on that run. If that's the case, gonna spitball some ideas

• Intelligently remove stuff from the prompt, ask only for a completion
• Transition to only asking for a diff

[jwmatthews]

Related to #137, it would help if we had a means to track the size of prompts/results to see if we are nearing limits.

[fabianvf]

I saw an example today where I think the response actually finished (it had the terminating ``` markers) but the file was still incomplete. Wonder if it's a quirk of the mixtral model? I don't think it would terminate the code block if it were cutting it off due to token limits

[jwmatthews]

I am seeing frequent partial responses from "mistralai/mixtral-8x7b-instruct-v01"

Below is a sample of console output where I am attempting to migrate 1 single file:
https://github.com/konveyor-ecosystem/coolstore/blob/main/src/main/java/com/redhat/coolstore/model/ShoppingCart.java

This appears to be a problematic file to attempt, I often see this file fail and I need to retry several times for success.

Below runs has 2 failures (partial output where it just stops when generating the updated source code)
3rd attempt succeeds.

https://gist.github.com/jwmatthews/e6ebc6a6f006b0e7bdbceeef59361137

The nature of this succeeding with retries makes me think this is not as direct as we've exceeded token size, yet I feel I am blind right now with tokens consumed in our request and our reply, it would help if we can get some metrics around token usage to gauge when we are getting close to a limit.