FAQ for using generative orchestration
These frequently asked questions (FAQ) describe the AI impact of the generative orchestration mode for custom copilots in Copilot Studio.
What is generative orchestration?
Generative orchestration lets your custom copilot answer user queries with relevant topics and/or actions. Generative orchestration enables more natural conversations by filling in inputs, using details from the conversation history. For example, if you ask about the nearest store in Kirkland, and then ask for the weather there, orchestration infers you want to ask for the weather in Kirkland. The system can also chain together multiple actions or topics. For example, it can answer "I need to get store hours and find my nearest store." When the copilot is unsure about details, it can ask followup questions to disambiguate.
What can generative orchestration do?
With generative orchestration, the system first creates a plan to answer the user query by using the name, description, inputs, and ouputs of the topics and actions available. It also references the last 10 turns of conversation history. It then tries to execute the plan by filling in required inputs from the conversation, following up with the user for any missing or ambiguous details. The system checks that it found an answer to the user's question before replying to the user. If not, it goes through the process again. Finally, the system generates a response based on the output of the plan from the topics and/or actions. It also uses any custom instructions for the copilot when generating the final response.
What are the intended uses of generative orchestration?
You can use this mode within your copilot to create a copilot that can answer user queries based on the conversation history, names and descriptions for topics, and names, descriptions, inputs, and outputs for actions.
How is generative orchestration evaluated? What metrics are used to measure performance?
Generative orchestration is evaluated for end-to-end quality at each step of the process. Quality is measured in terms of how well the system creates and executes a plan that successfully addresses the user query. Quality measure scores are labeled manually by our team during fine-tuning. We evaluate quality over a variety of user queries, prompts, and actions. We also evaluate how well the system does at ignoring malicious content from end users or plugin authors, and how well the system avoids producing harmful content.
What are the limitations of generative orchestration? How can users minimize the impact of generative orchestration limitations when using the system?
For best results, make sure your topics and actions include high quality descriptions. We provide guidance on how to write high quality descriptions in the Copilot Studio documentation.
What operational factors and settings allow for effective and responsible use of generative orchestration?
Generative orchestration is currently English only. Once you enable generative mode within your copilot, you can test the system to see how well it performs using the test panel. You can also add custom instructions for your copilot to help generate the final response.
What are actions and how does your copilot, with generative mode enabled, use them?
You can add actions to your custom copilot to answer user queries. You can use actions developed by Microsoft or third parties, or you can create your own actions. You configure which actions to configure for the custom copilot to use. You can also edit the name, description, inputs, and outputs used by the system.
What data can Copilot Studio provide to actions? What permissions do Copilot Studio actions have?
When your copilot calls an action, the action receives the input values specified by the action. The input values can include some of the conversation history with the end user.
What kinds of issues may arise when using Copilot Studio enabled with actions?
Actions may not always work as intended. Errors may occur when preparing the input for the action or when generating a response based on the action's output. Your copilot may also call the wrong action for the user query. To mitigate the risk of such errors when using actions, make sure you have high quality, relevant, and unambiguous descriptions configured for the actions in your custom copilot.