Confirming Responses

  • Article
  Microsoft Speech Technologies Homepage

This topic is the sixth of seven tutorial topics covering tools in the Microsoft Speech Application SDK Version 1.1 (SASDK). This section of the tutorial uses Speech Control Editor, which is the all-purpose tool for assembling a working speech-enabled Web application in Microsoft Visual Studio .NET 2003. Speech Control Editor manipulates ASP.NET Speech Controls that produce Speech Application Language Tags (SALT) and JScript on the client. The developer uses Speech Control Editor to specify prompts to speak to the user, to recognize answers using grammars, and to confirm the answers.

For reference information on Speech Control Editor, see Creating Speech-Enabled Web Pages for an introduction to the topic, and Recognizing and Confirming Answers for specifics on QA controls. In addition, Designing Dialogue Flow contains guidelines for designing voice-only applications.

This sequence of tutorials demonstrates how to build a simple voice-only ASP.NET Web application using the SASDK. Specifically, the tutorials demonstrate building a Start page of an imaginary pizza ordering service, for use by telephony Speech Application Language Tags (SALT) clients.

The procedures for creating the pizza ordering service application build on each other. Performing the procedures in sequence is therefore important.

Confirming the Responses

The application needs to confirm the responses that it has recognized to ensure that it has not recognized some phrase incorrectly, and to give the caller a chance to correct the error if it exists. In this application, if the caller indicates that the information is not correct, the dialogue flow returns to the first prompt for a response from the user, SizeQA, and repeats all the prompts again. The sample and reference applications included in the SASDK demonstrate other confirmation strategies, as well as other dialogue designs.

To add the confirmation QA control

  1. From the Speech tab on the Toolbox, drag a QA control onto PizzaPanel.
  2. Right-click the QA control, and select Property Builder.
  3. In the ID edit box, change the name of the control to ConfirmQA.
  4. Select Voice Output.
  5. Select the PlayOnce item.
    This specifies that the application will play this prompt only once while the tutorial application is first being tested.

To add semantic values to ConfirmQA

  1. In the tree view pane on the left, select Input.
  2. In the right pane, click Add Existing Grammar.
  3. In Add Existing Grammar, click Browse next to the Grammar Path edit box.
  4. In Grammar URL browser, click Grammars in the Projects: pane, and then double-click PizzaOrder.grxml in the Contents of 'Grammars' pane.
  5. In Add Existing Grammar, click Browse in the Active Rule section.
  6. In Rule Browser, select Confirm from the list of rules, and click Select Active Rule.
  7. In Add Existing Grammar, click OK.
  8. In ConfirmQA Properties, click XPathTrigger Sample Sentence Tool.
  9. In the Sample speech input edit box, type Yes, and click Go.
  10. On the Answers tab, double-click the first SemanticItem drop-down list box, and select siYes.
  11. Click the corresponding XPathTrigger drop-down list box, and select SML/Yes.
    That binds the semantic information in SML/Yes to the siYes SemanticItem in SemanticMap.
  12. In the Sample speech input edit box, now type No, and click Go.
  13. On the Answers tab, double-click the next SemanticItem drop-down list box, and select siNo.
  14. Click the corresponding XPathTrigger drop-down list box, and select SML/No.
    That binds the semantic information in SML/No to the siNo SemanticItem in SemanticMap.
  15. Click OK.
  16. Right-click the Default.aspx tab, and from the context menu, select Save Default.aspx.

Playing back the caller's responses as a prompt

To play back the caller's responses as a prompt, it is necessary to create a prompt function. This prompt function extracts the text from the semantic items that were filled when the caller responded to previous prompts. It takes the size and phone number text from the siSize, and the siPhoneNumberText, concatenates them with some phrases in the prompt database to produce a prompt that is not in the prompt database. This also requires a small function to extract the contents of the semantic items from the SemanticMap control. For more information on prompt functions, see Using Prompt Databases.

To create the prompt function file

  1. Right-click the ConfirmQA control, and select Property Builder.
  2. In the tree view pane on the left, click Voice Output.
  3. In the Prompt options section, clear the BargeIn check box.
    In this case, the caller needs to listen to the entire prompt in order to be able to decide if the order details and the telephone number are correct.
  4. Click the Prompt Function radio button.
  5. Click the link Manage this page's prompt function script files.
  6. On the Manage 'Default.aspx' Prompt Function Script Files dialog box, select New.
  7. On the New Prompt Function Script File dialog box, enter PizzaOrder into the File name edit box.
  8. Click Save, and then click OK.
  9. From the drop-down list under the Prompt Function radio button, select <New...>.
    The New Prompt Function dialog box appears, with PizzaOrder.pf indicated as the prompt function file and ConfirmQA_prompt as the prompt function.
  10. Click OK.

To add parameters for the ConfirmQA_prompt function

  1. On ConfirmQA Properties, click Edit prompt function file....

  2. On the message box that comes up, click OK.

  3. In the table of parameters, select the row under the History parameter and double-click the cell under Parameter Name.

  4. Type the following parameter name: sPhoneNumberText.

  5. Double-click the cell under Validation Value.

  6. In the Validation Values dialog box, enter "four two five five five five zero nine zero zero" (with the quote marks).
    In a pass through the solution to validate prompts, this value is used.

  7. Click OK.

  8. Double-click the cell under Runtime Value, and enter the following function call: GetSIValue(siPhoneNumberText).
    In a subsequent step, you'll write this function in the HTML on Default.aspx.

  9. Similarly, enter the following items in the table:

    Parameter Name Validation Values Runtime Value
    sSize "large"
    "medium"
    "small"    		 GetSIValue(siSize)

    When the table is complete, it looks like the following illustration.

prompt function parameters

  1. Copy the following function body, paste it into a plain text editor, copy it from the plain text editor and paste it into the function ConfirmQA_prompt_inner edit box, overwriting the default function body: 

    {
var sConfirmPromptStart = "You ordered a ";
var sConfirmPromptMid = " pizza, and your telephone number is ";
var sConfirmPromptEnd = " Is that correct?";
var sPrompt = "";
    

    sPrompt = sConfirmPromptStart + sSize + " " + sConfirmPromptMid;
sPrompt = sPrompt + sPhoneNumberText + sConfirmPromptEnd;
return(sPrompt);
}

Note  The spaces around the strings in the function body are important. If they are not present, the text in the strings is concatenated with the semantic items, and in most cases, the prompt database will not have recorded audio to match. In that case, the text-to-speech (TTS) engine plays the prompts.

  1. The entire Prompt Function pane now looks like the following illustration.

Prompt Function pane

  1. Right-click the PizzaOrder.pf tab, and from the context menu, select Save PizzaOrder.pf.

To retrieve the semantic values

The prompt function needs to retrieve the semantic items from the SemanticMap, and set its parameters equal to those values. The following Jscript function in the HTML on Default.aspx retrieves the value of the SemanticItem that it gets as a parameter. This new function is called by the ConfirmQA_prompt function, which was added in the previous section of this topic. It is enclosed in CDATA tags so the SALT interpreter does not try to interpret it.

  1. Click the Default.aspx tab.

  2. At the bottom of the design canvas, click HTML.
    The HTML view of Default.aspx opens in a source editor.

  3. Copy the following script element, paste it into a plain text editor, copy it from the plain text editor, and paste it into the source editor just above the opening <form> tag of the HTML. 

    <SCRIPT>
// <![CDATA[
function GetSIValue(theSI) {
return(theSI == null ? null : theSI.value);
}
// ]]>
</SCRIPT>

    The source editor now looks like the following illustration:

Script in HTML view

  1. Right-click the Default.aspx tab, and from the context menu, select Save Default.aspx.

To set the order of ConfirmQA

Dropping ConfirmQA onto the PizzaPanel put it last in the default execution order for added QA controls. ConfirmQA, however, should play before ThanksQA, so it needs to move up in the execution order.

  1. Click the Default.aspx tab.
  2. At the bottom of the design canvas, click Design.
    The Design canvas of Default.aspx opens.
  3. From the View menu, click Speech Controls Outline.
    The Speech Controls Outline shows the execution order of the controls in an application, and if expanded, shows additional features of the controls. In this case, for instance, some controls have Answers.
  4. In Speech Controls Outline, select ConfirmQA.
  5. At the top of the pane, click the up arrow once to move the control in the order between PhoneQA and ThanksQA.
    The Speech Controls Outline should now look like the following illustration.

Speech Controls Outline

To check the prompt playback

  1. On the Debug menu select Start. SASDK first builds the application, and then opens Speech Debugging Console and Telephony Application Simulator (TASim), with TASim on top.
  2. On TASim, click Dial. Through your headset or speakers, the application plays back the prompts in all the QA controls, one after the other, from the prompt database. To respond to prompts, on Speech Debugging Console click Use Audio and speak after Status indicates Listening. This time, the application asks for confirmation of the information provided.
  3. Close Speech Debugging Console and TASim.

Note  If the microphone is configured improperly, the results of this check may be unexpected, such as incomplete prompts, or missing information in the confirmation prompt. If this happens, it may be necessary to configure the microphone, as described in the section "Before Beginning the Tutorial," in Speech Application SDK Tutorial.

At this point, the prompt played back by the prompt function uses the voice produced by the TTS engine. The previous section recorded all the prompts, so something new has appeared in the prompts. The next section of the tutorial, Debugging Applications, demonstrates another method for debugging prompts, and determining what happened.

Handling Confirmations

If the response to the question, Is that correct? is Yes in response to the prompt from ConfirmQA, then the dialogue flow simply continues on to ThanksQA and the application terminates.

If the response, however, is No, the application needs to take different action. In this case, the dialogue flow begins again at the SizeQA prompt and continues through the prompts until the ConfirmQA prompt again. The onNotConfirmed function referred to in the OnClientChanged property of the siNo semantic item clears the values of the semantic items to which the QA controls refer. That starts the dialogue flow again from the first control in the speech index order with empty semantic items. In order to have the prompts play again if the caller does not confirm the information, it is first necessary to clear the PlayOnce item in properties of the QA controls.

For more information on the activation order of controls, see Activation and Semantic Processing.

To clear the PlayOnce properties

  1. Right-click SizeQA, and select Property Builder.
  2. In the tree view pane on the left, select Voice Output .
  3. Clear PlayOnce.
  4. Click OK.
  5. Repeat steps 1 through 4 for PhoneQA, and ConfirmQA.

To handle a negative confirmation

  1. Right-click SemanticMap, select Property Builder.

  2. In the tree view pane on the left, select siNo.

  3. In the OnClientChanged edit box, type onNotConfirmed.
    The script for this handler goes into the HTML markup on Default.aspx.

  4. Click OK.

  5. At the bottom of the design canvas, click HTML.
    The HTML view of Default.aspx opens in a source editor.

  6. Copy the following function body, paste it into a plain text editor, copy it from the plain text editor, and paste it into the source editor inside the SCRIPT tag element previously inserted, just above the closing tag of the CDATA element. 

    //Restarts the dialog if user doesn't confirm the information.
function onNotConfirmed()
{
if(siNo.value == "No")
{
siSize.Clear();
siPhoneNumberText.Clear();
siPhoneNumberAreaCodeDigits.Clear();
siPhoneNumberLocalDigits.Clear();
siYes.Clear();
siNo.Clear();
siCancel.Clear();
}
return;
}

    The function checks to ensure that the semantic value of siNo is "No," and if so, it clears the values of all the semantic items in the semantic map.
    The source editor now looks like the following illustration:

source editor

  1. Right-click the Default.aspx tab, and from the context menu, select Save Default.aspx.

Terminating the Application Properly

The application uses an AnswerCall control to make the proper connections to the telephony source. When the application terminates, it needs a DisconnectCall control to break those connections properly.

To use the DisconnectCall control

  1. Select the Default.aspx tab.
  2. Select Design at the bottom of the pane.
  3. On the Toolbox, select Speech, and then drag a DisconnectCall control onto the design canvas.
  4. Right-click the Default.aspx tab, and from the context menu, select Save Default.aspx.

Canceling the Order

Applications can use Command controls to listen to all speech input for a given scope of the application, and take action if a response is a recognized phrase in the control's grammar. Cancellation, Help and Navigation commands are common uses for these controls. See the sample Using Global Commands and Context-sensitive Help for more information. The tutorial application uses a simple control to cancel the order entirely and terminate the application.

To set up the Command control for order cancellation

  1. On the Toolbox, select Speech, and then drag a Command control onto the design canvas.

  2. Right-click Command1, and select Property Builder.

  3. Select Command in the treeview pane. In the Speech Settings Item drop-down list, select SpeechControlSettingsItem1.

  4. In the Scope drop-down list, select PizzaPanel.

  5. In the Type edit box, enter Cancel.

  6. In the tree view pane on the left, select Input.

  7. On the Grammar tab, set the grammar as an existing grammar, choosing PizzaOrder.grxml as the grammar file, and Cancel as the active rule.

  8. Click the XPathTrigger Sample Sentence Tool button, enter Cancel into the Sample speech input edit box, and click Go.
    The tool now shows the SML that the SR engine returned.

  9. In the Command trigger section, in the XPathTrigger drop-down list box, select SML/Cancel.

  10. In the Client Side Event section, in the OnClientCommand edit box, enter CancelFunction. The body of this function is added later in the tutorial.
    The script in this function sets the activation state of the QA controls to false, so they are not activated again.

  11. In the tree view pane on the left, select Voice Output.

  12. Click the Inline Prompt radio button, and enter the following inline prompt in the edit box:

    Your order has been canceled. Goodbye.

  13. Click OK.

To set the state of the QA controls

By default, if no client activation is specified for a control, its activation state is true. That means that it is activated any time it comes up in the dialogue flow, and its semantic items are not complete. In order to use the Command control to cancel the pizza order and terminate the application, it is necessary to set the state of all the QA controls to false. This requires a ClientActivation function for each control. It is the same for all the controls in this application, and initially sets the activation state of the control to true. CancelFunction sets the variable that the ClientActivation function returns to false. Then when the control becomes active in the dialogue flow, the ClientActivation function returns false, and the control does not activate, even though its semantic items may not be complete.

  1. Right-click SizeQA, and select Property Builder.
  2. In the Activation section, in the ClientActivationFunction edit box, type ActivateQA.

Note  The AllowCommands box is already checked by default. This allows Command controls to listen for responses and take action at the same time the QA control is active.

  1. Click OK.
  2. Repeat steps 1 through 3 for PhoneQA, ConfirmQA, and ThanksQA.

To insert the script

  1. At the bottom of the design canvas, click HTML.
    The HTML view of Default.aspx opens in a source editor.
  2. Copy the following function body, paste it into a plain text editor, copy it from the plain text editor, and paste it into the source editor inside the SCRIPT tag element previously inserted, just above the closing tag of the CDATA element. 
    // ActivateQA is the ClientActivationFunction for the QA controls
// Until the CancelFunction sets the value of activeQA to false,
// it is always true.
var activeQA=true;
function ActivateQA()
{
return activeQA;
}
function CancelFunction()
{
activeQA = false;
}
    The ActivateQA function returns the value of the variable activeQA. When the application starts, this variable is set to true. The CancelFunction sets that variable to false. After that, when the ActivateQA function is called, it returns true.
  3. Right-click the Default.aspx tab, and from the context menu, select Save Default.aspx.
    The design canvas should now look something like the following illustration. design canvas

Confirming the Dialogue Flow

The dialogue flow for the application is now complete. The application plays the prompts, waits for responses, replays the relevant responses, and confirms the responses. If the caller confirms the responses, the application plays a concluding prompt and terminates. If the caller does not confirm the responses, the application clears all the semantic items and restarts playing prompts with the SizeQA control. It is now possible to confirm the flow of the dialogue by running it in the debuggers, and answering Yes or No to the confirmation prompt.

Note  Because the PlayOnce property of QA controls is now cleared in order to be able to reactivate the controls on non-confirmation, the controls also now replay if the silence time-out is exceeded or if the speech recognition engine does not recognize a response. A production application would have counters for each prompt and a way to determine the confidence of the recognition. It could give a different prompt after an unrecognized response or it could change the dialogue flow after a certain number of failed attempts to get a recognizable response. For more information, see the samples Employing Confirmation and Correction Strategies and Responding to Silence and Unrecognized Input.

To confirm the dialogue flow

  1. On the Debug menu select Start.
  2. On TASim, click Dial.
  3. Respond to the prompts as they play.
  4. When asked if the responses are correct, respond No.
    The application should begin again with the prompt asking for the size.
  5. Again, respond to the prompts as they play.
  6. When asked if the responses are correct, respond Yes.
    The application should play its thanks prompt, and then terminate.
  7. Close Speech Debugging Console and TASim.
To See
Go to the next step Debugging Applications
Start from the beginning Creating a Speech Project
Get more information on prompts and prompt functions Using Prompt Databases
Playing a Prompt across Page Transitions
Constructing Natural-sounding Recorded Prompts
Get more information on commands Using Global Commands and Context-sensitive Help