How to extract similarity from Azure AI Search x Azure OpenAI completion

Hello Rohan Pathak,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

In summary, I understand that you are using Azure OpenAI's client.chat.completions.create method to carry out retrieval-augmented generation (RAG) by querying a vector database stored on Azure AI Search. You also need help configuring Azure AI Search or adjusting their request parameters to include similarity scores in the response.

Solution

To solve the problem of retrieving similarity scores from Azure AI Search and incorporating them into the Azure OpenAI completion responses, by following these examples, you should be able to:

1. Enable and configure similarity scoring in your Azure AI Search index. Modify your search queries to include the includeTotalResultCount and use queryType: "full". Here's how you can set up your search parameters to include scores:

   {
       "search": "*",
       "queryType": "full",
       "searchFields": "<field_names>",
       "select": "<field_names>",
       "includeTotalResultCount": true,
       "scoringProfile": null,
       "skip": 0,
       "top": 5
   }

2. Adjust the search parameters in your extra_body to include these scores.

This is to ensure your extra_body parameter in the OpenAI request correctly references the adjusted search parameters. The code sample below shows how to structure your extra_body to include the required search parameters:

extra_body = {
    "data_sources": [
        {
            "type": "azure_search",
            "parameters": {
                "endpoint": "<your_search_endpoint>",
                "index_name": "<your_index_name>",
                "key": "<your_search_key>",
                "search_parameters": {
                    "queryType": "full",
                    "includeTotalResultCount": True,
                    "searchFields": "<field_names>",
                    "select": "<field_names>"
                }
            }
        }
    ]
}

3. Parse the response from Azure OpenAI to extract and utilize the similarity scores.

The example below shows how you can handle the response and print the similarity scores along with the retrieved content:

import openai
# Initialize OpenAI client (ensure you have set up your API key and other configurations)
openai.api_key = "<your_openai_api_key>"
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Your system message"},
        {"role": "user", "content": "Your user query"}
    ],
    extra_body=extra_body
)
# Process the response to extract and display the similarity scores
for choice in response['choices']:
    if 'context' in choice:
        for context_item in choice['context']:
            title = context_item.get('title', 'N/A')
            text = context_item.get('text', 'N/A')
            score = context_item.get('score', 'N/A')  # Ensure the score is being retrieved
            print(f"Title: {title}\nText: {text}\nScore: {score}\n")

These modifications will ensure that your application can access and display the relevance of each retrieved chunk, enhancing the overall functionality and user experience.

References

To read more and see more step-by-steps in detail, kindly use the resources below. They provided more comprehensive guides on BM25 relevance scoring, vector relevance and ranking, re-ranking Cognitive Search results with Machine Learning, and Azure Vector Search:

Source: BM25 relevance scoring - Azure AI Search. Accessed, 5/31/2024.

Source: Vector relevance and ranking - Azure AI Search. Accessed, 5/31/2024.

Source: Re-ranking Cognitive Search results with Machine Learning. Accessed, 5/31/2024.

Source: Azure Vector Search: A new way of information - Medium. Accessed, 5/31/2024.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam