Create exact data match sensitive information type/rule package

Applies to

• Classic experience

You can create an exact data match (EDM) SIT (SIT) by using the Use the Exact Data Match schema and SIT pattern tool in the Microsoft Purview Compliance Portal, or you can create the rule package manually as an XML file. You can also combine the two methods by using one method to create the schema and later using the other method to edit it.

If you are not familiar with EDM-based SITS or their implementation, you should familiarize yourself with:

• Learn about sensitive information types
• Learn about exact data match based sensitive information types
• Get started with exact data match based sensitive information types

Prerequisites

Perform the steps in these articles:

1. Export source data for exact data match based sensitive information types
2. Create the schema for exact data match based sensitive information types
3. Hash and upload the sensitive information source table for exact data match sensitive information types

• Whether you will be creating an EDM SIT using the tool or the rule package XML file via PowerShell, you must have Global Administrator or Compliance Administrator permissions to create, test, and deploy a custom SIT through the UI. See About admin roles in Office 365.

• Identify one of the built-in SITs to use as the Primary elements SIT.
  • If none of the built-in SITs will match the data in the column you selected you will have to create a custom SIT that does.
  • If you selected the Ignored Delimiters option for the primary element column in your schema, make sure the custom SIT you create will match data with and without the selected delimiters.
  • If you use a built-in SIT, make sure it will detect exactly the strings you want to select, and not include any surrounding characters or exclude any valid part of the string as stored in your sensitive information table.

See Sensitive information type entity definitions and Create custom sensitive information types.

Use the Exact Data Match schema and SIT pattern tool

You can use this tool to create your SIT files to help simplify the process.

An EDM SIT is composed of one or more patterns. Each pattern describes a combination of fields from the schema that will be used to identify sensitive content in a document or email (evidence).

Select the appropriate tab for the portal you're using. To learn more about the Microsoft Purview portal, see Microsoft Purview portal. To learn more about the Compliance portal, see Microsoft Purview compliance portal.

Edit or delete a SIT pattern

Select the appropriate tab for the portal you're using. To learn more about the Microsoft Purview portal, see Microsoft Purview portal. To learn more about the Compliance portal, see Microsoft Purview compliance portal.

Working with specific types of data

For performance reasons, it is critical that you use patterns that will minimize the number of unnecessary matches. For example, you might use a SIT based on the regular expression.

\b\w*\b

This would match every individual word or number in any document or email. This would cause the service to be overloaded with matches and miss detecting true matches. Using more precise patterns can avoid this situation. Here are some recommendations for identifying the right configuration for some common types of data.

Email addresses: Email addresses can be easy to identify, but because they are so common in sensitive content, they might cause a significant load in the system if used as a primary field. Use email adderss only as secondary evidence. If they must be used as primary evidence, when you define your custom SIT use logic to exclude items where email addresses are used as From or To fields in emails. Also use logic to exclude email addresses from your company’s domain to reduce the number of unnecessary strings that need to be matched.

Phone numbers: Phone numbers can come in many different formats, including or excluding country/region prefixes, area codes, and separators. To reduce the false negatives while keeping load to a minimum, use them only as secondary elements, exclude all likely separators, like parenthesis and dashes and only include in your sensitive data table the part that will always be present in the phone number.

People's names: Don’t use people’s names as primary elements if using a SIT based on a regular expression as the classification element for this EDM type, because they are difficult to distinguish from common words.

If you must use a primary element that is hard to identify with a specific pattern (such as a project code name), that could generate a high volume of matches to be processed, make sure you include keywords in the SIT you use as the classification element for your EDM type. For example, if using project code names that are also regular words, you can use the word project as required additional evidence in close proximity to the project name regular expression-based pattern in the SIT you use as the classification element for your EDM type. Or, you might consider using a SIT based on a regular dictionary as the classification element for your EDM SIT.

When trying to match numeric strings, specify the allowed ranges of numbers such as the number of digits or the starting digits, if known. If you need to match a relatively flexible range of numbers, you can use keywords in the base SIT to reduce the number of matches. For example, if trying to match account numbers consisting of 7-11 digits, add the words account, customer, acct. to the SIT as required additional evidence. This reduces the likelihood of unnecessary matches that could result in exceeding the limits of EDM matches that can be processed.

If a field you need to use as a primary element follows a simple pattern that might result in large numbers of matches, and you can’t add the presence of keywords as additional evidence in the SIT, you can instead require a minimum number of occurrences of that pattern. For example, you could use a custom SIT defined in the following way to detect at least 29 other five-digit numbers surrounding a potential five-digit number to match against in your sensitive content:

 <Entity id="98703510-18b3-43d4-961f-15317594beb7"
                  patternsProximity="300"
                  recommendedConfidence="85"
                  relaxProximity="false">
                  <Pattern confidenceLevel="85"
                              proximity="300">
                              <IdMatch idRef="MRN"/>
                              <Match idRef="30 AccountNrs"
                                    minCount="30"
                                    proximity="3000"
                                    uniqueResults="true"/>
                  </Pattern>
      </Entity>
      <Regex id="30 AccountNrs">\d{5}</Regex>

In some cases, you might have to identify certain account or record identification numbers that for historical reasons don’t follow a standardized pattern. For example, Medical Record Numbers can be composed of many different permutations of letters and numbers within the same organization. Even though it might be hard at first to identify a pattern, closer inspection often lets you narrow down a pattern that describes all valid values without causing an excessive number of invalid matches. For example, it might be detected that “all MRNs are at least seven characters in length, have at least two numerical digits in them, and if they have any letters in them, they start with one”. Creating a regular expression based on such criteria should allow you to minimize unnecessary matches while capturing all the desired values, and further analysis might allow increased precision by defining separate patterns that describe different formats.

Create a rule package manually

This procedure shows you how to create a file in XML format called a rule package (with Unicode encoding), and then upload it into Microsoft Purview using Security & Compliance PowerShell cmdlets.

1. Create a rule package in XML format (with Unicode encoding), similar to the following example. (You can copy, modify, and use our example.)

   When you set up your rule package, make sure to correctly reference your .csv, .tsv, or pipe (|) delimited sensitive information source table file and edm.xml schema file. You can copy, modify, and use our example. In this sample xml the following fields need to be customized to create your EDM sensitive type:

   • RulePack id & ExactMatch id: Use New-GUID to generate a GUID.

   • Datastore: This field specifies EDM lookup data store to be used. You provide the data source name of the configured EDM Schema.

   • idMatch: This field points to the primary element for EDM.

   • Matches: Specifies the field to be used in exact lookup. You provide a searchable field name in EDM Schema for the DataStore.

   • Classification: This field specifies the SIT match that triggers EDM lookup. You can use the name or GUID of an existing built-in or custom SIT.

   Note

   Be aware that any string that matches the SIT provided will be hashed and compared to every entry in the sensitive information source table. To avoid performance issues if you choose a custom SIT for the classification element, don't use one that will match a large percentage of content. For example one that matches "any number" or "any five-letter word". You can differentiate it by adding supporting keywords or including formatting in the definition of the custom classification SIT.

   • Match: This field points to additional evidence found in proximity of idMatch.

   • Matches: You provide any field name in EDM Schema for DataStore.

   • Resource idRef: This section specifies the name and description for sensitive type in multiple locales

     • You provide GUID for ExactMatch ID.
     • Name & description: customize as required.

     <RulePackage xmlns="http://schemas.microsoft.com/office/2018/edm">
        <RulePack id="fd098e03-1796-41a5-8ab6-198c93c62b11">
          <Version build="0" major="2" minor="0" revision="0" />
          <Publisher id="eb553734-8306-44b4-9ad5-c388ad970528" />
          <Details defaultLangCode="en-us">
            <LocalizedDetails langcode="en-us">
              <PublisherName>IP DLP</PublisherName>
              <Name>Health Care EDM Rulepack</Name>
              <Description>This rule package contains the EDM sensitive type for health care sensitive types.</Description>
            </LocalizedDetails>
          </Details>
        </RulePack>
        <Rules>
          <ExactMatch id = "E1CC861E-3FE9-4A58-82DF-4BD259EAB371" patternsProximity = "300" dataStore ="PatientRecords" recommendedConfidence = "65" >
            <Pattern confidenceLevel="65">
              <idMatch matches = "SSN" classification = "U.S. Social Security Number (SSN)" />
            </Pattern>
            <Pattern confidenceLevel="75">
              <idMatch matches = "SSN" classification = "U.S. Social Security Number (SSN)" />
              <Any minMatches ="3" maxMatches ="6">
                <match matches="PatientID" />
                <match matches="MRN"/>
                <match matches="FirstName"/>
                <match matches="LastName"/>
                <match matches="Phone"/>
                <match matches="DOB"/>
              </Any>
            </Pattern>
          </ExactMatch>
          <LocalizedStrings>
            <Resource idRef="E1CC861E-3FE9-4A58-82DF-4BD259EAB371">
              <Name default="true" langcode="en-us">Patient SSN Exact Match.</Name>
              <Description default="true" langcode="en-us">EDM Sensitive type for detecting Patient SSN.</Description>
            </Resource>
          </LocalizedStrings>
        </Rules>
     </RulePackage>
     

2. Upload the rule package by running the following PowerShell command:

   New-DlpSensitiveInformationTypeRulePackage -FileData ([System.IO.File]::ReadAllBytes('.\\rulepack.xml'))
   

Next step

• For the Classic experience: Test an exact data match SIT