Azure ai search vs graphrag?

[HuskyDanny]

Hi Team,

I am very newbie at graphrag, I am using azure ai search for the basic RAG and I saw the general idea of graphrag is excellent and could be applied to all the private data we have, which creates the question "Should I abandon azure ai search and pursue graphrag for future data?".

Correct me if I were wrong. I am seeing this graphrag is a service that needs deploying ourselves, which only requires open ai services and storage service like azure storage account, and all the graph content is on the storage account and we will directly query on the service to get the relevant data. I am not sure what is the plan on azure ai search itself, but it seems to me like a decision to make for new integrations.

Any thought is appreciated!

[natoverse]

Thanks for a great question. I'll give some thoughts in two parts that I hope are helpful for you.

Conceptual

Azure AI Search is a very good option for standard RAG approaches. GraphRAG is suitable for certain types of queries on top of your private data, but comes with the tradeoff of higher cost and preparation time. For example, GraphRAG is particularly well-suited to high-level conceptual questions where an embedded query (such as "what are the top themes in my dataset") may not directly match any text fragments in a vector store to answer with, but GraphRAG's hierarchical summarization approach can provide an answer.

So you may think of these as complementary approaches, and depending on your domain and the types of questions your users need to ask, you may want one or both available.

Technical

GraphRAG local search does depend on using a vector store such as Azure AI Search in order to find the seed entities to launch graph traversal. We include support for LanceDB out of the box to streamline the evaluation process, but also support Azure AI Search for larger deployments. So you may want Azure AI Search alongside GraphRAG in your deployment anyway, even if you aren't using its RAG capabilities directly.

[HuskyDanny]

Thanks @natoverse, I like that you organized the reply into two parts.

Our scenario is nl2${X}ql, we would rely on the wiki of different flavors of sql. So conceptually, I think our scenario would benefit from "connecting the dots". Our users would always start with a high-level question and the application needs to find all its candidate docs. So I think graphrag could cover the traditional RAG for our case.

Technically, how the local search utilize the azure ai search? I didn't see steps about the Azure AI Search here .
Will the graphrag build the index of the sources also in the azure ai search for vector search? Since there are some steps to make things in azure ai search work like specifying the index schema, run indexer to extract data and build index... Can you please shed some lights into the scene behind?

[HuskyDanny]

This actually brings me another thought, I have put it in the feature request : #915, but let me know if that makes sense, Thanks!

[natoverse]

Yeah, we are missing good docs for how to configure a vector store (#690)

[csanadpoda]

Thanks for a great question. I'll give some thoughts in two parts that I hope are helpful for you.

Conceptual

Azure AI Search is a very good option for standard RAG approaches. GraphRAG is suitable for certain types of queries on top of your private data, but comes with the tradeoff of higher cost and preparation time. For example, GraphRAG is particularly well-suited to high-level conceptual questions where an embedded query (such as "what are the top themes in my dataset") may not directly match any text fragments in a vector store to answer with, but GraphRAG's hierarchical summarization approach can provide an answer.

So you may think of these as complementary approaches, and depending on your domain and the types of questions your users need to ask, you may want one or both available.

Technical

GraphRAG local search does depend on using a vector store such as Azure AI Search in order to find the seed entities to launch graph traversal. We include support for LanceDB out of the box to streamline the evaluation process, but also support Azure AI Search for larger deployments. So you may want Azure AI Search alongside GraphRAG in your deployment anyway, even if you aren't using its RAG capabilities directly.

I'm a bit confused about this @natoverse. I've been reading the paper on arxiv, and there it's stated:

Given a user query, the community summaries generated in the previous step can be used to generate
a final answer in a multi-stage process.
For a given community level, the global answer to any user query is generated as follows:

1. Prepare community summaries. Community summaries are randomly shuffled and divided
   into chunks of pre-specified token size. This ensures relevant information is distributed
   across chunks, rather than concentrated (and potentially lost) in a single context window.
2. Map community answers. Generate intermediate answers in parallel, one for each chunk.
   The LLM is also asked to generate a score between 0-100 indicating how helpful the generated answer is in answering the target question. Answers with score 0 are filtered out.
3. Reduce to global answer. Intermediate community answers are sorted in descending order
   of helpfulness score and iteratively added into a new context window until the token limit
   is reached. This final context is used to generate the global answer returned to the user.

So, first of all, I don't see where AI Search comes into the picture. Based on this, all the community descriptions are chunked and shuffled and then passed to an LLM. And second of all, there's no graph traversal, there's just a chosen level of the graph and all the community descriptions are used within that level. Even in the Evaluation section, you evaluate it on different levels of the communities and summarize descriptions per community level, there's no graph traversal involved.

I've also been wondering why an Azure AI Search service is needed, but your answer just confused me more instead of clearing things up. What am I missing?

It's even stated in the paper that you don't rely on traversal:

In contrast with related work that exploits the structured retrieval
and traversal affordances of graph indexes (subsection 4.2), we focus on a previously unexplored
quality of graphs in this context: their inherent modularity (Newman, 2006) and the ability of community detection algorithms to partition graphs into modular communities of closely-related nodes.