Are Azure OpenAI Service Rate Limits Shared Between Deployments?

Hi Arik Levy,

Thanks for Reaching the Microsoft Q&A Forum.

Each deployment has its own limits, which can affect how requests are processed. In your case, the dev service has a TPM of 10k and a corresponding RPM limit that allows for approximately 60 requests per minute

Even though you have implemented a one-second delay between requests in your loop, the service may still be evaluating the request rate over shorter intervals (like 1 or 10 seconds). If you send multiple requests within these intervals, you could exceed the RPM limit.

 Each deployment (dev and prod) has its own rate limits. However, if both services share the same subscription or resource group, they may be subject to a global quota that could potentially lead to interference. This means that high usage on one deployment could impact the other if they share limits at the subscription level.

Verify whether both deployments share any resources or quotas that could be leading to conflicts in rate limiting. If they do, consider restructuring your deployments to isolate their usage.

Thank you!

Both deployments are in different subscriptions. I'm just running the code below:

My gpt-4 deployment (I set the TPM low to test this) is at 10k TPM and 60 RPM Why am I hitting a 429 Too Many Requests, error by the 15th request? My initial thought was that my other gpt-4 deployment in prod was the culprit but they don't share a subscription.

for i in range(60):
    try:
        # Send the request to the client
        response = client.chat.completions.create(
            model=deployment_name,
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": 'What is the capital of Spain?'}
            ],
            temperature=0
        )
        # Log the count of requests
        logging.info("Request %d: Successful", i + 1)
        time.sleep(2)

        total_tokens += response.usage.total_tokens
        print(total_tokens)

Hi Arik Levy,

The error 429 means that you have submitted too many tokens or requests in a short period of time and have exceeded the number of requests allowed.

Azure services often enforce quotas on transactions per minute (TPM). If your application or service exceeds this quota, it will start receiving 429 errors.

Sometimes, 429 errors can occur due to transient issues while the Azure OpenAI service scales up to meet demand. In such cases, implementing retry logic with exponential backoff can help mitigate the issue.

If you have multiple deployments, ensure that the workload is evenly distributed among them. Sharp changes in workload can trigger rate limits, so gradually increasing the workload can help avoid hitting these limits.

Thank you!