Imagine you’re at a bookstore looking for the perfect book. You want recommendations that are not only on your favorite genre but also varied enough to introduce you to new authors. Retrieval-Augmented Generation systems work similarly by combining the strengths of finding relevant information and generating creative responses. To measure how well these systems perform, we use metrics like Hit Rate, which checks how often the right recommendations show up, and Mean Reciprocal Rank (MRR), which looks at the order of those recommendations. Maximum Marginal Relevance (MMR) helps ensure that the suggestions are both relevant and diverse. By using these metrics, we can make sure that the recommendations are not just accurate but also varied and interesting.
Hit Rate is one of the measures used to assess how well recommendation systems work. The desired item’s frequency of appearance in the top-N recommendations is measured. Within the framework of RAG, Hit Rate denotes the frequency with which pertinent data is successfully included into the output that is produced.
The calculation of Hit Rate involves dividing the total number of queries by the frequency with which the pertinent item appears in the top-N recommendations. In terms of math, it is stated as:
Let’s get a better understanding with an example. We have three queries Q1, Q2, Q3. We also know the exact node to be picked for those queries. Actual Nodes for those queries are N1, N2, N3. Now upon sending those queries we receive nodes from our Retriever. The retrieved nodes for those queries are as mentioned below:
We can see that our retriever has retrieved the correct node for Q1 and Q2. It did not perform well with Q3. Hence the Hit Rate is 1 for Q1, Q2 and 0 for Q3. Upon using our formula we can calculate the Hit Rate:
Now that we understand the Hit Rate metric to evaluate our model. We will now see the challenges faced using Hit Rate as our evaluation metric.
The major challenge that we face when using Hit Rate as our evaluation metric is that it does not take into account the position of the retrieved node. To understand it more, let’s see an example. Let’s say we have two retrievers – retriever 1 and retriever 2. And below image shows the retrieved nodes by both the retrievers.
From the above image we can see that both the retrievers have retrieved the correct node for Q1 and Q2 but not Q3. Hence they both get the same hit rate percentage.
But when inspecting them further we can see that retriever 1 has retrieved the correct node of Q1 at position three and retriever 2 has retrieved the correct node of Q1 at position one. Hence retriever 2 should get a higher score than retriever 1, but the Hit Rate does not take the position of retrieved nodes into account. Now here is where the new metric MRR (Mean Reciprocal Rank) comes into picture.
One statistical metric used to assess an information retrieval system’s efficacy is Mean Reciprocal Rank (MRR). It is especially helpful in situations where a query is answered by the system returning a ranked list of things (like documents or answers). MRR is used to evaluate the retrieval component of the system’s performance in retrieving pertinent documents that facilitate the development of accurate and pertinent responses in the context of Retrieval-Augmented development (RAG).
N: Number of queries, ranki is the rank position of the first relevant document for the i-th query.
Let’s see an example for MRR.
In the above image we can see that MRR for Q1 is ⅓ as the correct retrieved node is at 3rd position. Hence the MRR is calculated as
We can see that while the Hit Rate is 66.66% still the MRR is at 44.4% and for retrievers retrieving correct nodes at starting positions get more weightage.
Maximum Marginal Relevance (MMR) re-ranks results to enhance both their relevance and diversity. In order to guarantee that the items returned are both relevant and sufficiently varied to address all facets of the query, MMR attempts to strike a balance between novelty and relevance.
Here, D is the set of all candidate documents, R is the set of already selected documents, q is the query, Sim1 is the similarity function between a document and the query, and Sim2 is the similarity function between two documents. di and dj are documents in D and R respectively.
The parameter λ (mmr_threshold) controls the trade-off between relevance (the first term) and diversity (the second term). When the mmr_threshold is close to 1, the system prioritizes relevance; when it is close to 0, it prioritizes diversity.
Let’s look into a simple example that illustrates MMR. We will use the same example as Hit Rate to demonstrate how MMR re-ranks the retrieved nodes.
To proceed with MMR let’s assume some variables like Relevance Score:
Similarity Score:
For simplicity, let’s set λ=0.5\lambda = 0.5λ=0.5 to give equal weight to relevance and diversity.
The Maximum Marginal Relevance (MMR) is calculated by re-ranking retrieved documents to balance relevance and diversity, ensuring a relevant and varied list of results.
Final order for Q1: [N1,N3,N2]
Final order for Q2: [N3,N1,N5]
Final order for Q3: [N1,N4,N2]
Using MMR, we re-rank the nodes to ensure a balance between relevance and diversity. The final re-ranked nodes are:
Metrics like Hit Rate, Mean Reciprocal Rank and Maximal Marginal Relevance (MMR) are essential for assessing and improving the effectiveness of RAG systems. While MMR maintains a balance between relevance and diversity in the recovered results, Hit Rate, MRR concentrates on the frequency of retrieving pertinent information. RAG systems can greatly increase the calibre and applicability of the responses they create, which will increase user happiness and confidence, by optimizing these metrics.
A. We determine it by dividing the total number of searches by the number of hits, or relevant items, in the top-N. We determine it by dividing the total number of searches by the number of hits, or relevant items, in the top-N.
A. A re-ranking technique called Maximum Marginal Relevance (MMR) strikes a balance between the relevance and diversity of items obtained. By taking into account a document’s relevance to the query and how similar it is to previously selected items, it seeks to decrease redundancy.
A. In RAG systems, the Hit Rate—a measure of the frequency with which pertinent information is retrieved—is essential for producing precise and contextually relevant replies. Better success in retrieving relevant information is indicated by a greater hit rate.
A. MMR minimises redundancy by ensuring that the collection of recovered documents is both diverse and pertinent. This facilitates the provision of thorough answers that address all facets of the inquiry.
Data science intern at Analytics Vidhya, specializing in ML, DL, and AI. Dedicated to sharing insights through articles on these subjects. Eager to learn and contribute to the field's advancements. Passionate about leveraging data to solve complex problems and drive innovation.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
