This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 2%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
I have an Azure OpenAI deployment used by multiple internal users that charges back based on token usage found in the "usage" field of the API response. However, users who stream the response with "stream=True" do not receive the "usage" field in the Azure OpenAI response. Is there any way to retrieve the token count even with "stream=True"?.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 11%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Thanks for the question, Here is the sample for token count for stream enabled. Jupyter notebooks to calculate tokens usage with Tiktoken for scenarios with and without Token Streaming. https://github.com/LazaUK/AOAI-Streaming-TokenUsage/tree/main
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 44%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHW%3C/text%3E%3C/svg%3E)
This is about guessing the token usage. What we all need is the usage object that we get when we don't use streaming. Why is it so hard to return that with the final block(s)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 74%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWE%3C/text%3E%3C/svg%3E)
This doesn't answer the question. It should be an API response similar to what OpenAI are offering today.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 66%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELR%3C/text%3E%3C/svg%3E)
That is not an appropriate response, calculating tokens is inexact. Many models on Azure are currently lacking to return the appropriate token usage.
Here are some models I've tested on Azure that do not return the appropriate token usage count when streaming responses.
ChatGPT 4 turbo
ChatGPT 4o
All Llama models including 405b
I noticed that the stream of GPT4o mini has a usage event object whren you add the right parameters to the api call. For some reason GPT4o does not.