Automated feedback loop

[luarss]

Think about an automatic mechanism in which we can use the feedback we get from the users back in our evaluation dataset.

Use it as part of our dataset ground truth (good or bad )

Tasks

• DeepEval using explicit feedback https://docs.confident-ai.com/docs/confident-ai-human-feedback

For this, we need to institute more metrics (tracked in #18 )

Explicit feedback

• Thumbs up/ thumbs down. But this is rare.

Implicit feedback

• Inferred from user behavior, such as response time, follow-up questions, or even actions like copying and pasting responses. Implicit feedback is far more abundant and can provide valuable insights into how well the LLM is performing

1. https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7
2. https://www.nebuly.com/blog/llm-feedback-loop

[Kannav02]

Hey @luarss!

how would you want the feedback to be stored, do you want a feeder pipeline, that feeds back into the evaluation dataset and then based on that each subsequent prompt answer is influenced by these metrics

do you also want the thumbs up and thumbs down feature to be implemented, I believe the functionality would be similar to how LLMs like chatGPT take the feedback, like once about 100 prompts maybe?

If those are the requirements can I work on this issue?

thank you!