Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models
We introduce a novel method, Rowen, a novel framework that enhances LLMs with a adaptive retrieval augmentation process tailored to address hallucinated outputs. This process is governed by a consistency-based uncertainty estimation module, which assesses the model's uncertainty regarding the input query by estimating semantic inconsistencies across different languages and models. Upon detecting high uncertainty on input questions, Rowen activates the retrieval of external information to rectify the model outputs.
- openai == 0.28.0
- Serper Api Key
- OpenAI Api Key
- Run experiments on the TruthfulQA dataset
python run_truthfulqa.py --k K --threshold THRESHOLD --alpha ALPHA --qwen_model_name qwen-max-0428 --mode hybrid- Run experiments on the StrategyQA dataset
python run_strategyqa.py --k K --threshold THRESHOLD --alpha ALPHA --qwen_model_name qwen-max-0428 --mode hybridGenerated responses and evaluation results will be saved automatically.
Please be careful, the experiments are relatively expensive because Rowen calls OpenAI API multiple times for a single question. You can decrease the k (number of pertubed questions) in the file to reduce the cost.
