Strange Behavior with AssistantAPI: Missing Files and Annotation References using Azure OpenAI 4o

Luca Pierpaoli 26 Reputation points
2024-06-17T09:42:38.7233333+00:00

I'm encountering a strange behavior using the AssistantAPI. Although the assistant successfully responds with information about the asset created, it does not provide the expected files and annotation references for download in the content response as documented here and here

The issue does not occur when using GPT-4 128K Turbo, where the expected response format is correctly returned with the file and annotation references.

Expected Response:


{

  "id": "XXX",
  "assistant_id": "YYY",
  "content": [
  {
   "image_file": {
   "file_id": "assistant-1YGVTvNzc2JXajI5JU9F0HMD"
  },
  "type": "image_file"
}]

Actual Response: 

{
  "id": "XXX",
  "assistant_id": "YYY",
  "attachments": null,
  "completed_at": null,
  "content": [
    {
      "text": {
        "annotations": [],
        "value": "Your document is ready and you can download here '<a href=\"/mnt/data/your_doc.docx\">download</a>"
      },
      "type": "text"
    }
  ],
}

Analyzing the response, I noticed that there are no values or new references for retrieving assets (images or annotations). The attachments field is null, and the file_ids array is empty.

Has anyone experienced similar issues or have insights on why the annotations and file_ids are missing in the response? Any suggestions on how to resolve this discrepancy?

Thank you!

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,536 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
2,577 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. mohmdRzvani-603709138563430 0 Reputation points
    2024-06-17T11:33:25.6766667+00:00

    {

    "text": {

    "annotations": [],

"value": "Your document is ready and you can download here '<a href=\"/mnt/data/your_doc.docx\">download</a>"

    },

    "type": "text"

    }

  2. Luca Pierpaoli 26 Reputation points
    2024-06-19T12:48:00.2366667+00:00

    The root cause is quite intriguing.

    When migrating to 4o, we modified the assistant instructions by adding the following phrase at the end:

    "Format the answer as HTML. Here example: omitted example for brevity"

    It appears that this instruction causes the model to respond with the HTML inside the "value" property. However, for some unknown reasons, it removes the references to "annotations" and "images" from the response.

    I was able to replicate this issue on the OpenAI Assistant (non-Azure) and observed the same behavior.

    Upon removing the final prompt, the functionality returned to normal.