Is there a parameter to get multiple replicates of an answer from Phi3?

Angelo D Ambrosio 0

Hello,

I'm trying to use a Phi3 model via curl API. The "n" parameter, which on OpenAI models asks the model to generate n replicates of the answer doesn't work with the Phi models. Is there an alternative or it simply doesn't implement it?

I also tried to parallelize the querying, but it gave a 500 error on 5 parallel requests.

Thanks

YutongTie-MSFT 51,696 Reputation points

2024-06-13T23:42:37.57+00:00

Thanks for reaching out to us, where do you develop your model? Any document you are following so that we can reproduce your issue?

It seems the Phi3 model does not support the "n" parameter for generating multiple replicates of an answer like other OpenAI models. It's possible that this feature hasn't been implemented yet or may not be supported due to the architecture or design choices specific to the Phi3 model.

Parallelizing queries can sometimes lead to server errors like the 500 error you encountered, especially if there's a high load on the server or if the server has limits on the number of concurrent requests it can handle. It's generally a good idea to avoid making too many parallel requests to the API to prevent overwhelming the server.
Angelo D Ambrosio 0 Reputation points

2024-06-17T11:13:08.86+00:00

Indeed, I noticed that with a largish context, Phi3 medium 128k gets very slow, much slower than much bigger models like Mistral Large, while being served in the same way (serverless) from Azure AI. It's a pity since I was counting on Phi3 for a fast, long-context, medium-accuracy model.
Furthermore, with around ~ 20k tokens Phi3 just crashes with error 500. I guess not enough memory was allocated.

This is the specific model page: https://ai.azure.com/explore/models/Phi-3-medium-128k-instruct/version/1/registry/azureml?tid=6ad73702-409c-4046-ae59-cc4bea334507

2 answers

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

Is there a parameter to get multiple replicates of an answer from Phi3?

2 answers

Your answer