Azure OpenAI Service frequently asked questions

Azure OpenAI doesn't use customer data to retrain models. For more information, see the Azure OpenAI data, privacy, and security guide.

Our current APIs allow up to 10 custom headers, which are passed through the pipeline, and returned. We have noticed some customers now exceed this header count resulting in HTTP 431 errors. There is no solution for this error, other than to reduce header volume. In future API versions we will no longer pass through custom headers. We recommend customers not depend on custom headers in future system architectures.

Azure OpenAI is supported by the latest release of the OpenAI Python library (version>=1.0). However, it's important to note migration of your codebase using openai migrate is not supported and will not work with code that targets Azure OpenAI.

GPT-4 Turbo Preview is the gpt-4 (1106-preview) model. To deploy this model, under Deployments select model gpt-4. For Model version select 1106-preview. To check which regions this model is available, refer to the models page.

Azure OpenAI supports the latest GPT-4 models. It supports both GPT-4 and GPT-4-32K.

Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-3, Codex, and DALL-E models with the security and enterprise promise of Azure. Azure OpenAI codevelops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.

With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI.

Yes, as part of Azure AI services, Azure OpenAI supports VNETs and Private Endpoints. To learn more, consult the Azure AI services virtual networking guidance.

No, GPT-4 is designed by OpenAI to be multimodal, but currently only text input and output are supported.

Previously, the process for adding new use cases required customers to reapply to the service. Now, we're releasing a new process that allows you to quickly add new use cases to your use of the service. This process follows the established Limited Access process within Azure AI services. Existing customers can attest to any and all new use cases here. Note that this is required anytime you would like to use the service for a new use case you didn't originally apply for.

This error typically occurs when you try to send a batch of text to embed in a single API request as an array. Currently Azure OpenAI only supports arrays of embeddings with multiple inputs for the text-embedding-ada-002 Version 2 model. This model version supports an array consisting of up to 16 inputs per API request. The array can be up to 8,191 tokens in length when using the text-embedding-ada-002 (Version 2) model.

Check out our introduction to prompt engineering. While these models are powerful, their behavior is also very sensitive to the prompts they receive from the user. This makes prompt construction an important skill to develop. After you've completed the introduction, check out our article on system messages.

This is expected behavior when using the default sign-in experience for the Azure AI Studio.

To access Azure AI Studio from a guest account that has been granted access to an Azure OpenAI resource:

1. Open a private browser session and then navigate to https://ai.azure.com.
2. Rather than immediately entering your guest account credentials instead select Sign-in options
3. Now select Sign in to an organization
4. Enter the domain name of the organization that granted your guest account access to the Azure OpenAI resource.
5. Now sign-in with your guest account credentials.

You should now be able to access the resource via the Azure AI Studio.

Alternatively if you're signed into the Azure portal from the Azure OpenAI resource's Overview pane you can select Go to Azure AI Studio to automatically sign in with the appropriate organizational context.

Azure OpenAI models (including GPT-4) being unable to correctly identify what model is running is expected behavior.

Why does this happen?

Ultimately, the model is performing next token prediction in response to your question. The model doesn't have any native ability to query what model version is currently being run to answer your question. To answer this question, you can always go to Azure AI Studio > Management > Deployments > and consult the model name column to confirm what model is currently associated with a given deployment name.

The questions, "What model are you running?" or "What is the latest model from OpenAI?" produce similar quality results to asking the model what the weather will be today. It might return the correct result, but purely by chance. On its own, the model has no real-world information other than what was part of its training/training data. In the case of GPT-4, as of August 2023 the underlying training data goes only up to September 2021. GPT-4 wasn't released until March 2023, so barring OpenAI releasing a new version with updated training data, or a new version that is fine-tuned to answer those specific questions, it's expected behavior for GPT-4 to respond that GPT-3 is the latest model release from OpenAI.

If you wanted to help a GPT based model to accurately respond to the question "what model are you running?", you would need to provide that information to the model through techniques like prompt engineering of the model's system message, Retrieval Augmented Generation (RAG) which is the technique used by Azure OpenAI on your data where up-to-date information is injected to the system message at query time, or via fine-tuning where you could fine-tune specific versions of the model to answer that question in a certain way based on model version.

To learn more about how GPT models are trained and work we recommend watching Andrej Karpathy's talk from Build 2023 on the state of GPT.

This is expected behavior. The models aren't able to answer questions about themselves. If you want to know when the knowledge cutoff for the model's training data is, consult the models page.

This is expected behavior. First there's no guarantee that every recent event was part of the model's training data. And even when information was part of the training data, without using additional techniques like Retrieval Augmented Generation (RAG) to help ground the model's responses there's always a chance of ungrounded responses occurring. Both Azure OpenAI's use your data feature and Bing Chat use Azure OpenAI models combined with Retrieval Augmented Generation to help further ground model responses.

The frequency that a given piece of information appeared in the training data can also impact the likelihood that the model will respond in a certain way.

Asking the latest GPT-4 Turbo Preview model about something that changed more recently like "Who is the prime minister of New Zealand?", is likely to result in the fabricated response Jacinda Ardern. However, asking the model "When did Jacinda Ardern step down as prime minister?" Tends to yield an accurate response which demonstrates training data knowledge going to at least January of 2023.

So while it is possible to probe the model with questions to guess its training data knowledge cutoff, the model's page is the best place to check a model's knowledge cutoff.

Legacy pricing information is available via a downloadable PDF file. For all other models, consult the official pricing page.

You can minimize the occurrence of these errors by reducing the temperature of your prompts to less than 1 and ensuring you're using a client with retry logic. Reattempting the request often results in a successful response.

If the service performs processing, you will be charged even if the status code is not successful (not 200). Common examples of this are, a 400 error due to a content filter or input limit, or a 408 error due to a timeout. Charges will also occur when a status 200 is received with a finish_reason of content_filter. In this case the prompt did not have any issues, but the completion generated by the model was detected to violate the content filtering rules, which result in the completion being filtered. If the service doesn't perform processing, you won't be charged. For example, a 401 error due to authentication or a 429 error due to exceeding the Rate Limit.

A Limited Access registration form is not required to access most Azure OpenAI models. Learn more on the Azure OpenAI Limited Access page.

For monthly updates, see our what's new page.

Check out our introduction to Azure OpenAI training course.

• We recommend posting questions on Microsoft Q&A.
• Alternatively, you can post questions on Stack Overflow.

Azure OpenAI is part of Azure AI services. You can learn about all the support options for Azure AI services in the support and help options guide.

Consult the Azure OpenAI model availability guide.

Consult the Azure OpenAI model availability guide for region availability.

We do offer an Availability SLA for all resources and a Latency SLA for Provisioned-Managed Deployments. For more information about the SLA for Azure OpenAI Service, see the Service Level Agreements (SLA) for Online Services page.

In order to successfully access fine-tuning, you need Cognitive Services OpenAI Contributor assigned. Even someone with high-level Service Administrator permissions would still need this account explicitly set in order to access fine-tuning. For more information, please review the role-based access control guidance.

A base model is a model that hasn't been customized or fine-tuned for a specific use case. Fine-tuned models are customized versions of base models where a model's weights are trained on a unique set of prompts. Fine-tuned models let you achieve better results on a wider number of tasks without needing to provide detailed examples for in-context learning as part of your completion prompt. To learn more, review our fine-tuning guide.

100

If a customized (fine-tuned) model is deployed for more than 15 days during which no completions or chat completions calls are made to it, the deployment is automatically deleted (and no further hosting charges are incurred for that deployment). The underlying customized model remains available and can be redeployed at any time. To learn more, check out the how-to-article.

There are currently two different REST APIs that allow model deployment. For the latest model deployment features such as the ability to specify a model version during deployment for models like text-embedding-ada-002 Version 2, use the Deployments - Create Or Update REST API call.

No, quota Tokens-Per-Minute (TPM) allocation isn't related to the max input token limit of a model. Model input token limits are defined in the models table and aren't impacted by changes made to TPM.

No, we don't support fine-tuning the image capabilities of GPT-4 at this time.

No, you can use dall-e-3 to generate images and gpt-4-vision-preview to understand images.

We currently support PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), and nonanimated GIF (.gif).

Yes, we restrict image uploads to 20 MB per image.

No, we'll delete the image for you automatically after it has been processed by the model.

We process images at the token level, so each image we process counts towards your tokens per minute (TPM) limit. See the Image tokens section of the Overview for details on the formula used to determine token count per image.

No, the model doesn't receive image metadata.

If an image is ambiguous or unclear, the model will do its best to interpret it. However, the results might be less accurate. A good rule of thumb is that if an average human can't see the info in an image at the resolutions used in low/high res mode, then the model can't either.

See the limitations section of the GPT-4 Turbo with Vision concepts guide.

By default GPT-4 vision-preview and GPT-4 turbo-2024-04-09 have a max_tokens value of 16. Depending on your request this value is often too low and can lead to truncated responses. To resolve this issue, pass a larger max_tokens value as part of your chat completions API requests. GPT-4o defaults to 4096 max_tokens.

Yes. Unlike Chat Completions API, Azure OpenAI Assistants is a stateful API, meaning it retains data. There are two types of data stored in the Assistants API:

• Stateful entities: Threads, messages, and runs created during Assistants use.
• Files: Uploaded during Assistants setup or as part of a message.

Data is stored in a secure, Microsoft-managed storage account that is logically separated.

All used data persists in this system unless you explicitly delete this data. Use the delete function with the thread ID of the thread you want to delete. Clearing the Run in the Assistants Playground does not delete threads, however deleting them using delete function will not list them in the thread page.

No. Currently Assistants supports only local files uploaded to the Assistants-managed storage. You cannot use your private storage account with Assistants.

Today we support CMK for Threads and Files in Assistants. See the What's new page for available regions for this feature.

No. Data is not used for Microsoft not used for training models. See the Responsible AI documentation for more information.

Azure OpenAI Assistants endpoints are regional, and data is stored in the same region as the endpoint. For more information, see the Azure data residency documentation.

• Inference cost (input and output) of the base model you're using for each Assistant (for example gpt-4-0125). If you've created multiple Assistants, you will be charged for the base model attached to each Assistant.
• If you've enabled the Code Interpreter tool. For example if your assistant calls Code Interpreter simultaneously in two different threads, this would create two Code Interpreter sessions, each of which would be charged. Each session is active by default for one hour, which means that you would only pay this fee once if your user keeps giving instructions to Code Interpreter in the same thread for up to one hour.
• File search is billed based on the vector storage used.

For more information, see the pricing page.

No. All quotas apply to using models with Assistants.

Assistants API only supports Azure OpenAI models.

The Assistants API is currently in public preview. Stay informed of our latest product updates by regularly visiting our What's New page.

See the Conceptual, quickstart, how-to articles for information on getting started and using Assistants. You can also check out Azure OpenAI Assistants code samples on GitHub.

You can customize your published web app in the Azure portal. The source code for the published web app is available on GitHub, where you can find information on changing the app frontend, as well as instructions for building and deploying the app.

Your app code won't be overwritten when you update your app. The app will be updated to use the Azure OpenAI resource, Azure AI Search index (if you're using Azure OpenAI on your data), and model settings selected in the Azure AI Studio without any change to the appearance or functionality.

Azure OpenAI on your data is a feature of the Azure OpenAI Services that helps organizations to generate customized insights, content, and searches using their designated data sources. It works with the capabilities of the OpenAI models in Azure OpenAI to provide more accurate and relevant responses to user queries in natural language. Azure OpenAI on your data can be integrated with customer's existing applications and workflows, offers insights into key performance indicators, and can interact with users seamlessly.

All Azure OpenAI customers can use Azure OpenAI on your data via the Azure AI studio and Rest API.

Azure OpenAI on your data supports ingestion from Azure AI Search, Azure Blob Storage, and uploading local files. You can learn more about Azure OpenAI on your data from the conceptual article and quickstart.

When using Azure OpenAI on your data, you incur costs when you use Azure AI Search, Azure Blob Storage, Azure Web App Service, semantic search and OpenAI models. There's no additional cost for using the "your data" feature in the Azure AI Studio.

You can prepare the index yourself using a script provided on GitHub. Using this script will create an Azure AI Search index with all the information needed to better use your data, with your documents broken down into manageable chunks. See the README file with the data preparation code for details on how to run it.

You can schedule an automatic index refresh, or upload additional data to your Azure Blob Container and use it as your data source when you create a new index. The new index will include all of the data in your container.

See Using your data for more information on supported file types.

Yes, Azure OpenAI on your data is part of the Azure OpenAI Service and works with the models available in Azure OpenAI. The content filtering and abuse monitoring features of Azure OpenAI still apply. For more information, see the overview of Responsible AI practices for Azure OpenAI models and the Transparency Note for Azure OpenAI for extra guidance on using Azure OpenAI on your data responsibly.

Yes, the token limit on the system message is 400. If the system message is more than 400 tokens, the rest of the tokens beyond the first 400 will be ignored. This limitation only applies to the Azure OpenAI on your data feature.

Azure OpenAI on your data currently doesn't support function calling.

You must send queries in the same language of your data. Your data can be in any of the languages supported by Azure AI Search.

When you select "Azure AI Search" as the data source, you can choose to apply semantic search. If you select "Azure Blob Container" or "Upload files" as the data source, you can create the index as usual. Afterwards you would reingest the data using the "Azure AI Search" option to select the same index and apply Semantic Search. You will then be ready to chat on your data with semantic search applied.

When you select "Azure Blob Container", "Azure AI Search", or "Upload files" as the data source, you can also select an Ada embedding model deployment to use when ingesting your data. This will create an Azure AI Search index with vector embeddings.

Index creation can fail when adding embeddings to your index if the rate limit on your Ada embedding model deployment is too low, or if you have a very large set of documents. You can use this script provided on GitHub to create the index with embeddings manually.

The Customer Copyright Commitment is a provision to be included in the December 1, 2023, Microsoft Product Terms that describes Microsoft’s obligation to defend customers against certain third-party intellectual property claims relating to Output Content. If the subject of the claim is Output Content generated from the Azure OpenAI Service (or any other Covered Product that allows customers to configure the safety systems), then to receive coverage, customer must have implemented all mitigations required by the Azure OpenAI Service documentation in the offering that delivered the Output Content. The required mitigations are documented here and updated on an ongoing basis. For new services, features, models, or use cases, new CCC requirements will be posted and take effect at or following the launch of such service, feature, model, or use case. Otherwise, customers will have six months from the time of publication to implement new mitigations to maintain coverage under the CCC. If a customer tenders a claim, the customer will be required to demonstrate compliance with the relevant requirements. These mitigations are required for Covered Products that allow customers to configure the safety systems, including Azure OpenAI Service; they do not impact coverage for customers using other Covered Products.

Next steps

• Azure OpenAI quotas and limits
• Azure OpenAI what's new
• Azure OpenAI quickstarts