Why does the chat completion API response stop early when using a custom content filter/data?

Duncan Eley 0 Reputation points
2024-06-25T16:13:31.2666667+00:00

When streaming responses using the chat completion API the answers complete as expected (within max_tokens).

However, if I add a custom content filter (with Streaming mode of Asynchronous Filter) to the model deployment, messages will stop short with a finish_reason value of stop.

If I change the content filter to use the Default streaming mode, the answers are complete, as expected.

More info:

  • This only appears to be an issue when using custom data in Azure AI Search.
  • Model name: gpt-4
  • Model version: 0613
  • I can replicate this problem in the Azure OpenAI Chat Playground.

Example request:

{
    "messages": [
        {
            "role": "user",
            "content": "initial question"
        },
        {
            "role": "assistant",
            "content": "initial response"
        },
        {
            "role": "user",
            "content": "follow up"
        }
    ],
    "temperature": 0.0,
    "max_tokens": 1000,
    "top_p": 1.0,
    "stop": null,
    "stream": true,
    "model": "my-deployment",
    "extra_body": {
        "data_sources": [
            {
                "type": "azure_search",
                "parameters": {
                    "endpoint": "https://my-search-endpoint.search.windows.net",
                    "authentication": {
                        "type": "api_key",
                        "api_key": "*****"
                    },
                    "index_name": "my-search-index",
                    "fields_mapping": {
                        "content_fields": [],
                        "title_field": null,
                        "url_field": null,
                        "filepath_field": null,
                        "vector_fields": []
                    },
                    "in_scope": false,
                    "top_n_documents": 5,
                    "query_type": "simple",
                    "semantic_configuration": "",
                    "role_information": "my role info",
                    "filter": null,
                    "strictness": 3
                }
            }
        ]
    }
}
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,536 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
2,577 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sina Salam 6,581 Reputation points
    2024-06-25T20:23:37.84+00:00

    Hello Duncan Eley,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    Problem

    I understand that you are having issue with custom filter in Asynchronous mode.

    Solution

    The issue you're encountering with the custom content filter in streaming mode using Asynchronous Filter seems to be related to how the filter interacts with the streaming process and the integration with Azure AI Search.

    You can check the following to resolve the issue:

    • Review your custom content filter's logic is compatible with streaming mode. Asynchronous filtering might be causing a delay or interruption in the stream, leading to the premature stop.
    • Incrementally test each part of your request and filter logic to identify the exact cause. This includes trying different configurations of the data source and filter settings.
    • Also, ensure all components (model, content filter, Azure AI Search) are up-to-date with the latest patches and updates from Azure.
    • Lastly, use the example below to adjust your request to see if simplifying or changing certain parameters helps.
    {
    "messages": [
        {
            "role": "user",
            "content": "initial question"
        },
        {
            "role": "assistant",
            "content": "initial response"
        },
        {
            "role": "user",
            "content": "follow up"
        }
    ],
    "temperature": 0.0,
    "max_tokens": 500,  // Reduce max tokens for testing
    "top_p": 1.0,
    "stop": null,
    "stream": true,
    "model": "my-deployment",
    "extra_body": {
        "data_sources": [
            {
                "type": "azure_search",
                "parameters": {
                    "endpoint": "https://my-search-endpoint.search.windows.net",
                    "authentication": {
                        "type": "api_key",
                        "api_key": "*****"
                    },
                    "index_name": "my-search-index",
                    "fields_mapping": {
                        "content_fields": [],
                        "title_field": null,
                        "url_field": null,
                        "filepath_field": null,
                        "vector_fields": []
                    },
                    "in_scope": false,
                    "top_n_documents": 5,
                    "query_type": "simple",
                    "semantic_configuration": "",
                    "role_information": "my role info",
                    "filter": null,
                    "strictness": 3
                }
            }
        ]
    }
}

    Accept Answer

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.

    ** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

    Best Regards,

    Sina Salam

    0 comments