README.md

Awesome Information Retrieval in the Age of Large Language Model

A curated list of awesome papers about information retrieval (IR) in the age of large language model (LLM). These include retrieval augmented large language model, large language model for information retrieval, and so on. If I missed any papers, feel free to open a PR to include them! And any feedback and contributions are welcome!

This list is currently maintained by Yinqiong Cai, Yu-An Liu, Shiyu Nee, and Hengran Zhang at CAS Key Lab of Network Data Science and Technology, ICT, CAS.

We thank all the great contributors very much.

Contents

• Survey

  • Retrieval Augmented LLM
  • LLM for IR

• Perspective Papers

• Retrieval Augmented LLM

  • For Pre-training LLM
  • For Fine-tuning LLM
  • For Inference of LLM
  • Joint Optimization of IR and LLM

• LLM for IR

  • Generating Synthetic Queries
  • Generating Synthetic Documents
  • Generating Ranking Lists
  • Query Understanding
  • Query Extension
  • Generate rather than Retrieve

• LLM for IR

• Benchmark and Evaluation

• Toolkits

Survey

Retrieval Augmented LLM

• Retrieval-based Language Models and Applications Akari Asai et.al. ACL 2023. (Tutorial)
• Augmented Language Models: a Survey Grégoire Mialon et.al. Arxiv 2023.
• Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community Qingyao Ai et.al. Arxiv 2023.
• Retrieval-Augmented Generation for Large Language Models: A Survey. Yunfan Gao et.al. Arxiv 2023.
• A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models. Yujuan Ding et.al. Arxiv 2024.

LLM for IR

• Large Language Models for Information Retrieval: A Survey Yutao Zhu et.al. Arxiv 2023.

Perspective Papers

• Perspectives on Large Language Models for Relevance Judgment Guglielmo Faggioli et.al. ICTIR 2023. (Best paper)
• Information Retrieval Meets Large Language Models. Zheng Liu et.al. WWW 2024.

Retrieval Augmented LLM

For Pre-training LLM

• REALM: Retrieval augmented language model pre-training. Kelvin Guu et.al. ICML 2020.
• Improving language models by retrieving from trillions of tokens. Sebastian Borgeaud et.al. ICML 2022. (RETRO)
• Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. Boxin Wang et.al. Arxiv 2023.

For Fine-tuning LLM

• Dense Passage Retrieval for open-domain question answering. Vladimir Karpukhin et.al. EMNLP 2020. (DPR)
• RAG: Retrieval-augmented generation for knowledge-intensive NLP tasks. Patrick Lewis et.al. NeurIPS 2020.
• FiD: Leveraging passage retrieval with generative models for open domain question answering. Gautier Izacard, Edouard Grave EACL 2021.
• Copy Is All You Need. Tian Lan et.al. ICLR 2023. (COG)
• Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models. Wenhao Yu et.al. Arxiv 2023. (Chain-of-Note)

For Inference of LLM

• Generalization through memorization: Nearest neighbor language models. Urvashi Khandelwal et.al. Arxiv 2019.
• Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. Harsh Trivedi et.al. Arxiv 2022.
• Rethinking with retrieval: Faithful large language model inference. Hangfeng He et.al. Arxiv 2023.
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation Ruiyang Ren et.al. Arxiv 2023.
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. Alex Mallen et.al. ACL 2023.
• When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation Shiyu Ni et.al. Arxiv 2024.

Joint Optimization of IR and LLM

• Atlas: Few-shot Learning with Retrieval Augmented Language Models. Gautier Izacard et.al. Arxiv 2022.
• REPLUG: Retrieval-Augmented Black-Box Language Models. Weijia Shi et.al. Arxiv 2023.
• Learning to Retrieve In-Context Examples for Large Language Models. Liang Wang et.al. Arxiv 2023.

LLM for IR

Generating Synthetic Queries

• InPars: Data augmentation for information retrieval using large language models. Luiz Bonifacio et.al. SIGIR 2022.
• UPR: Improving passage retrieval with zero-shot question generation. Devendra Singh Sachan et.al. EMNLP 2022.
• Promptagator: Fewshot dense retrieval from 8 examples. Zhuyun Dai et.al. ICLR 2023.

Generating Synthetic Documents

• Precise Zero-Shot Dense Retrieval without Relevance Labels. Luyu Gao et.al. Arxiv 2022.
• Generating Synthetic Documents for Cross-Encoder Re-Rankers: A Comparative Study of ChatGPT and Human Experts. Arian Askari et.al. Arxiv 2023.

Generating Ranking Lists

• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. Weiwei Sun et.al. Arxiv 2023.
• Zero-Shot Listwise Document Reranking with a Large Language Model. Xueguang Ma et.al. Arxiv 2023.
• Are Large Language Models Good at Utility Judgments? Hengran Zhang et.al. SIGIR 2024.
• RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs Yue Yu et.al. Arxiv 2024.
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy Hengran Zhang et.al. Arxiv 2024.

Query Understanding

• Query Understanding in the Age of Large Language Models. Avishek Anand et.al. Gen-IR 2023.

Query Extension

• Generative relevance feedback with large language models. Iain Mackie et.al. Arxiv 2023.
• Query2doc: Query expansion with large language models. Liang Wang et.al. Arxiv 2023.
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy Zhihong Shao et al. findings of EMNLP 2023

Generate rather than Retrieve

• Generate rather than retrieve: Large language models are strong context generators. Wenhao Yu et.al. ICLR 2023.

Bias for IR

• LLMs may Dominate Information Access: Neural Retrievers are Biased Towards LLM-Generated Texts. Sunhao Dai et.al. Arxiv 2023.

Benchmark and Evaluation

• KILT: a benchmark for knowledge intensive language tasks. Fabio Petroni et.al. NAACL 2021.

Toolkits

• RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit. Jiongnan Liu et.al. Arxiv 2023. RETA-LLM

Note

• Shiyu is currently focusing on the timing for triggering RAG (when to retrieve) based on the responses from the LLMs. Regarding the papers, you can visit https://github.com/ShiyuNee/Awesome-Calibration-Papers