Text Analysis Authoring - Get Model Evaluation Summary
Gets the evaluation summary of a trained model. The summary includes high level performance measurements of the model e.g., F1, Precision, Recall, etc.
GET {Endpoint}/language/authoring/analyze-text/projects/{projectName}/models/{trainedModelLabel}/evaluation/summary-result?api-version=2023-04-01
URI Parameters
Name | In | Required | Type | Description |
---|---|---|---|---|
Endpoint
|
path | True |
string |
Supported Cognitive Services endpoint (e.g., https://.api.cognitiveservices.azure.com). |
project
|
path | True |
string |
The name of the project to use. |
trained
|
path | True |
string |
The trained model label. |
api-version
|
query | True |
string |
Client API version. |
Responses
Name | Type | Description |
---|---|---|
200 OK | TextAnalysisAuthoringEvaluationSummary: |
List of all evaluation results. |
Other Status Codes |
Error response. |
Security
AADToken
These are the Azure Active Directory OAuth2 Flows. When paired with Azure role-based access control it can be used to control access to Azure Maps REST APIs. Azure role-based access controls are used to designate access to one or more Azure Maps resource account or sub-resources. Any user, group, or service principal can be granted access via a built-in role or a custom role composed of one or more permissions to Azure Maps REST APIs.
To implement scenarios, we recommend viewing authentication concepts. In summary, this security definition provides a solution for modeling application(s) via objects capable of access control on specific APIs and scopes.
Notes
- This security definition requires the use of the
x-ms-client-id
header to indicate which Azure Maps resource the application is requesting access to. This can be acquired from the Maps management API.
The Authorization URL
is specific to the Azure public cloud instance. Sovereign clouds have unique Authorization URLs and Azure Active directory configurations.
*
The Azure role-based access control is configured from the Azure management plane via Azure portal, PowerShell, CLI, Azure SDKs, or REST APIs.
*
Usage of the Azure Maps Web SDK allows for configuration based setup of an application for multiple use cases.
- Currently, Azure Active Directory v1.0 or v2.0 supports Work, School, and Guests but does not support Personal accounts.
Type:
oauth2
Flow:
implicit
Authorization URL:
https://login.microsoftonline.com/common/oauth2/authorize
Scopes
Name | Description |
---|---|
https://cognitiveservices.azure.com/.default | https://cognitiveservices.azure.com/.default |
Ocp-Apim-Subscription-Key
A subscription key for a Language service resource.
Type:
apiKey
In:
header
Examples
Successful Get Model Evaluation Summary
Sample request
GET {Endpoint}/language/authoring/analyze-text/projects/LoanAgreements/models/model2/evaluation/summary-result?api-version=2023-04-01
Sample response
{
"projectKind": "CustomEntityRecognition",
"customEntityRecognitionEvaluation": {
"confusionMatrix": {
"BorrowerAddress": {
"BorrowerAddress": {
"normalizedValue": 86.206894,
"rawValue": 3.4482758
},
"$none": {
"normalizedValue": 13.793103,
"rawValue": 0.55172414
}
},
"BorrowerCity": {
"BorrowerCity": {
"normalizedValue": 100,
"rawValue": 4
}
},
"BorrowerName": {
"BorrowerName": {
"normalizedValue": 100,
"rawValue": 4
}
},
"BorrowerState": {
"BorrowerState": {
"normalizedValue": 100,
"rawValue": 4
}
},
"Date": {
"Date": {
"normalizedValue": 100,
"rawValue": 4
}
},
"Interest": {
"Interest": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LenderAddress": {
"LenderAddress": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LenderCity": {
"LenderCity": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LenderName": {
"LenderName": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LenderState": {
"LenderState": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LoanAmountNumbers": {
"LoanAmountNumbers": {
"normalizedValue": 100,
"rawValue": 4
}
},
"LoanAmountWords": {
"LoanAmountWords": {
"normalizedValue": 100,
"rawValue": 4
}
},
"$none": {
"$none": {
"normalizedValue": 99.81485,
"rawValue": 51.90372
},
"BorrowerAddress": {
"normalizedValue": 0.18315019,
"rawValue": 0.0952381
},
"Interest": {
"normalizedValue": 0.002005294,
"rawValue": 0.0010427529
}
}
},
"entities": {
"Date": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"BorrowerName": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"BorrowerAddress": {
"f1": 0.6666666865348816,
"precision": 0.6000000238418579,
"recall": 0.75,
"truePositiveCount": 3,
"trueNegativeCount": 0,
"falsePositiveCount": 2,
"falseNegativeCount": 1
},
"BorrowerCity": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"BorrowerState": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LenderName": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LenderAddress": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LenderCity": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LenderState": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LoanAmountWords": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"LoanAmountNumbers": {
"f1": 1,
"precision": 1,
"recall": 1,
"truePositiveCount": 4,
"trueNegativeCount": 0,
"falsePositiveCount": 0,
"falseNegativeCount": 0
},
"Interest": {
"f1": 0.75,
"precision": 0.75,
"recall": 0.75,
"truePositiveCount": 3,
"trueNegativeCount": 0,
"falsePositiveCount": 1,
"falseNegativeCount": 1
}
},
"microF1": 0.94845366,
"microPrecision": 0.93877554,
"microRecall": 0.9583333,
"macroF1": 0.9513889,
"macroPrecision": 0.9458334,
"macroRecall": 0.9583333
},
"evaluationOptions": {
"kind": "percentage",
"trainingSplitPercentage": 80,
"testingSplitPercentage": 20
}
}
Definitions
Name | Description |
---|---|
Error |
The error object. |
Error |
Human-readable error code. |
Error |
Error response. |
Evaluation |
Represents the evaluation kind. By default, the evaluation kind is set to percentage. |
Inner |
Human-readable error code. |
Inner |
An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses. |
Project |
The project kind. |
Text |
Represents the evaluation summary for a custom entity recognition project. |
Text |
Represents the evaluation summary for a custom multi-label classification project. |
Text |
Represents the evaluation summary for a custom single-label classification project. |
Text |
Represents the evaluation summary for an entity. |
Text |
Represents the evaluation summary for a custom entity recognition project. |
Text |
Represents the options used running the evaluation. |
Text |
Represents the evaluation summary of a class in a multi-label classification project. |
Text |
Represents the evaluation summary for a multi-label classification project. |
Text |
Represents the evaluation summary for a class in a single-label classification project. |
Text |
Represents the evaluation summary for a custom single-label classification project. |
Error
The error object.
Name | Type | Description |
---|---|---|
code |
One of a server-defined set of error codes. |
|
details |
Error[] |
An array of details about specific errors that led to this reported error. |
innererror |
An object containing more specific information than the current object about the error. |
|
message |
string |
A human-readable representation of the error. |
target |
string |
The target of the error. |
ErrorCode
Human-readable error code.
Name | Type | Description |
---|---|---|
AzureCognitiveSearchIndexLimitReached |
string |
|
AzureCognitiveSearchIndexNotFound |
string |
|
AzureCognitiveSearchNotFound |
string |
|
AzureCognitiveSearchThrottling |
string |
|
Conflict |
string |
|
Forbidden |
string |
|
InternalServerError |
string |
|
InvalidArgument |
string |
|
InvalidRequest |
string |
|
NotFound |
string |
|
OperationNotFound |
string |
|
ProjectNotFound |
string |
|
QuotaExceeded |
string |
|
ServiceUnavailable |
string |
|
Timeout |
string |
|
TooManyRequests |
string |
|
Unauthorized |
string |
|
Warning |
string |
ErrorResponse
Error response.
Name | Type | Description |
---|---|---|
error |
The error object. |
EvaluationKind
Represents the evaluation kind. By default, the evaluation kind is set to percentage.
Name | Type | Description |
---|---|---|
manual |
string |
Split the data according to the chosen dataset for every example in the data. |
percentage |
string |
Split the data into training and test sets according to user-defined percentages. |
InnerErrorCode
Human-readable error code.
Name | Type | Description |
---|---|---|
AzureCognitiveSearchNotFound |
string |
|
AzureCognitiveSearchThrottling |
string |
|
EmptyRequest |
string |
|
ExtractionFailure |
string |
|
InvalidCountryHint |
string |
|
InvalidDocument |
string |
|
InvalidDocumentBatch |
string |
|
InvalidParameterValue |
string |
|
InvalidRequest |
string |
|
InvalidRequestBodyFormat |
string |
|
KnowledgeBaseNotFound |
string |
|
MissingInputDocuments |
string |
|
ModelVersionIncorrect |
string |
|
UnsupportedLanguageCode |
string |
InnerErrorModel
An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.
Name | Type | Description |
---|---|---|
code |
One of a server-defined set of error codes. |
|
details |
object |
Error details. |
innererror |
An object containing more specific information than the current object about the error. |
|
message |
string |
Error message. |
target |
string |
Error target. |
ProjectKind
The project kind.
Name | Type | Description |
---|---|---|
CustomEntityRecognition |
string |
For building an extraction model to identify your domain categories using your own data. |
CustomMultiLabelClassification |
string |
For building a classification model to classify text using your own data. Each file can have one or many labels. For example, file 1 is classified as A, B, and C and file 2 is classified as B and C. |
CustomSingleLabelClassification |
string |
For building a classification model to classify text using your own data. Each file will have only one label. For example, file 1 is classified as A and file 2 is classified as B. |
TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary
Represents the evaluation summary for a custom entity recognition project.
Name | Type | Description |
---|---|---|
customEntityRecognitionEvaluation |
Contains the data related to extraction evaluation. |
|
evaluationOptions |
Represents the options used running the evaluation. |
|
projectKind |
string:
Custom |
Represents the project type that the evaluation ran on. |
TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary
Represents the evaluation summary for a custom multi-label classification project.
Name | Type | Description |
---|---|---|
customMultiLabelClassificationEvaluation |
Text |
Contains the data related to multi label classification evaluation. |
evaluationOptions |
Represents the options used running the evaluation. |
|
projectKind |
string:
Custom |
Represents the project type that the evaluation ran on. |
TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary
Represents the evaluation summary for a custom single-label classification project.
Name | Type | Description |
---|---|---|
customSingleLabelClassificationEvaluation |
Text |
Contains the data related to single label classification evaluation. |
evaluationOptions |
Represents the options used running the evaluation. |
|
projectKind |
string:
Custom |
Represents the project type that the evaluation ran on. |
TextAnalysisAuthoringEntityEvaluationSummary
Represents the evaluation summary for an entity.
Name | Type | Description |
---|---|---|
f1 |
number |
Represents the model precision |
falseNegativeCount |
integer |
Represents the count of false negative |
falsePositiveCount |
integer |
Represents the count of false positive |
precision |
number |
Represents the model recall |
recall |
number |
Represents the model F1 score |
trueNegativeCount |
integer |
Represents the count of true negative |
truePositiveCount |
integer |
Represents the count of true positive |
TextAnalysisAuthoringEntityRecognitionEvaluationSummary
Represents the evaluation summary for a custom entity recognition project.
Name | Type | Description |
---|---|---|
confusionMatrix |
object |
Represents the confusion matrix between two entities (the two entities can be the same). The matrix is between the entity that was labelled and the entity that was predicted. |
entities |
Represents the entities evaluation |
|
macroF1 |
number |
Represents the macro F1 |
macroPrecision |
number |
Represents the macro precision |
macroRecall |
number |
Represents the macro recall |
microF1 |
number |
Represents the micro F1 |
microPrecision |
number |
Represents the micro precision |
microRecall |
number |
Represents the micro recall |
TextAnalysisAuthoringEvaluationOptions
Represents the options used running the evaluation.
Name | Type | Description |
---|---|---|
kind |
Represents the evaluation kind. By default, the evaluation kind is set to percentage. |
|
testingSplitPercentage |
integer |
Represents the testing dataset split percentage. Only needed in case the evaluation kind is percentage. |
trainingSplitPercentage |
integer |
Represents the training dataset split percentage. Only needed in case the evaluation kind is percentage. |
TextAnalysisAuthoringMultiLabelClassEvaluationSummary
Represents the evaluation summary of a class in a multi-label classification project.
Name | Type | Description |
---|---|---|
f1 |
number |
Represents the model precision |
falseNegativeCount |
integer |
Represents the count of false negative |
falsePositiveCount |
integer |
Represents the count of false positive |
precision |
number |
Represents the model recall |
recall |
number |
Represents the model F1 score |
trueNegativeCount |
integer |
Represents the count of true negative |
truePositiveCount |
integer |
Represents the count of true positive |
TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary
Represents the evaluation summary for a multi-label classification project.
Name | Type | Description |
---|---|---|
classes |
<string,
Text |
Represents the classes evaluation |
macroF1 |
number |
Represents the macro F1 |
macroPrecision |
number |
Represents the macro precision |
macroRecall |
number |
Represents the macro recall |
microF1 |
number |
Represents the micro F1 |
microPrecision |
number |
Represents the micro precision |
microRecall |
number |
Represents the micro recall |
TextAnalysisAuthoringSingleLabelClassEvaluationSummary
Represents the evaluation summary for a class in a single-label classification project.
Name | Type | Description |
---|---|---|
f1 |
number |
Represents the model precision |
falseNegativeCount |
integer |
Represents the count of false negative |
falsePositiveCount |
integer |
Represents the count of false positive |
precision |
number |
Represents the model recall |
recall |
number |
Represents the model F1 score |
trueNegativeCount |
integer |
Represents the count of true negative |
truePositiveCount |
integer |
Represents the count of true positive |
TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary
Represents the evaluation summary for a custom single-label classification project.
Name | Type | Description |
---|---|---|
classes |
<string,
Text |
Represents the classes evaluation |
confusionMatrix |
object |
Represents the confusion matrix between two classes (the two classes can be the same). The matrix is between the class that was labelled and the class that was predicted. |
macroF1 |
number |
Represents the macro F1 |
macroPrecision |
number |
Represents the macro precision |
macroRecall |
number |
Represents the macro recall |
microF1 |
number |
Represents the micro F1 |
microPrecision |
number |
Represents the micro precision |
microRecall |
number |
Represents the micro recall |