The metadata file (model.json) for the Common Data Model

The metadata file (or model.json) in a Common Data Model folder describes the data in the folder, metadata and location, as well as how the file was generated and by which data producer.

The model file enables:

• Discoverability based on data-producer metadata
• Obtaining semantic information about entity records / attributes and links to underlying data files
• Indicates compliance with Common Data Model standard entities
• Information about when entities were most recently updated

Download the schema of the model.json file.

Model-file contents

The Common Data Model folder contents are described by a standardized metadata .json file that contains these key categories of metadata:

• Root elements

  Metadata that applies to all entities within the folder, such as description, last modified time, and data culture

• Entity information

  • General metadata such as type, description, etc.

  • Attributes, including metadata such as data types

  • [optional] Compliance with Common Data Model standard entity schema shapes

    For example, a given entity in the folder comply with all the attributes required for a ‘base’ Account entity

  • Data file (partition) locations, and metadata about data encoding of data files

• Reference models

  Existing entities and datafiles to which this model.json file refers

• Relationships

  Describes the relationships among the entities

In addition to the standard elements, additional producer-specific elements or extensions can be added with a prefix, such as “pbi:mashup” where “pbi:” is the prefix. Consumers can ignore these fields, but the libraries will roundtrip the fields to ensure no loss of fidelity.

The following sections describes each standard category in more detail.

Root properties

These properties are listed at the root of the model.json file and describe properties relevant to all entities, relationships, etc.

Sample:

{
   "application":"MyApplication",
   "name":"OrdersProducts",
   "description":"Model containing data for Order and Products.",
   "version":"1.0",
   "modifiedTime":"2018-01-02T12:00:00+08:00",
   "annotations":[

   ],
   "entities":[

   ],
   "referenceModels":[

   ]
}

Annotations

Property name: annotations

Optional, non-essential contextual information (key/value pairs) that can be used to store additional context about a property in the model file. Annotations can be applied at multiple levels including to entities and attributes. Producers can add a prefix, such as “pbi:MappingDisplayHint” where “pbi:” is the prefix, when annotations aren't necessarily relevant to other consumers.

Sample:

{
   "annotations":[
      {
         "name":"MyApp:VersionNum",
         "value":"3.0.1"
      }
   ]
}

Reference Models

Property name: referenceModels

Reference models are existing model.json files with entities and datafiles to which that this model.json file refers. This allows existing metadata and data to be reused without copying it into this Common Data Model folder.

Sample:

{
   "referenceModels":[
      {
         "id":"f19bbb97-c031-441a-8bd1-61b9181c0b83 ",
         "location":"https://contosostorageaccount.dfs.core.windows.net/contosoFilesystem/cdmFolder/model.json"
      }
   ]
}

Entities

Property name: entities

An entity is a collection of attributes and metadata that defines a concept (such as Account or Contact) and can be defined by any data producer. An entity may match the schema of a standard entity, defined as part of the Common Data Model and listed in the GitHub repository. Entities that don’t match one of the standard shapes are said to be custom entities.

In the model.json file, you can specify local or reference entities. A local entity has the details about the entity (such as attributes, schemas, and partitions) defined within the model.json file, and the data files are expected to be within the same Common Data Model folder. A referenced entity is pointer to an entity defined by a referenced model, where all the metadata and data files are defined.

Depending on the type of entity, you may find different fields defined:

R - required, O - optional, (blank) – n/a

Entity-level properties

In the required columns, “n/a” means the field shouldn’t exist for that type of entity.

LocalEntity sample:

{
   "entities":[
      {
         "$type":"LocalEntity",
         "name":"Products",
         "description":"Information about products and their pricing information.",
         "annotations":[

         ],
         "attributes":[

         ],
         "partitions":[

         ],
         "schemas":[
            "https://raw.githubusercontent.com/Microsoft/CDM/master/schemaDocuments/core/applicationCommon/foundationCommon/Product.0.7.cdm.json"
         ]
      }
   ]
}

ReferencedEntity sample:

{
   "entities":[
      {
         "$type":"ReferenceEntity",
         "name":"Products",
         "description":"Information about products and their pricing information.",
         "source":"Products",
         "modelId":"f19bbb97-c031-441a-8bd1-61b9181c0b83",
         "annotations":[

         ]
      }
   ]
}

Attributes

Property name: attributes

Attributes are the fields within an entity that correspond to data values within the data file. For example, a Contact entity may have attributes such as First Name and Last Name.

Sample:

{
   "attributes":[
      {
         "name":"productId",
         "description":"The unique identifier of the product.",
         "dataType":"string",
         "annotations":[

         ]
      }
   ]
}

Partitions

Property name: partitions

The partition array indicates the name and location of the actual data files that correspond to the entity definition. Today, the partition contains a full URI to the location of the data file, so new data files must be manually added to the list of partitions.

File Format Settings

Property name: fileFormatSettings

The file-format settings provide metadata related to the data files in the partition. Depending on the type, you may find different fields defined, but only one type is supported today. Encoding of the data files is UTF-8, but you can request other encodings by creating an issue in the GitHub repository.

Relationships

Property name: relationships

Relationships describe how entities are connected, such as an Account that has a Contact. As of this writing, only relationships based on a single attribute in the source and target entities is supported, but more types might be supported in the future.

ReferenceAttribute

Sample:

{
   "relationships":[
      {
         "$type":"SingleKeyRelationship",
         "fromAttribute":{
            "entityName":"Account",
            "attributeName":"ContactId"
         },
         "toAttribute":{
            "entityName":"Contact",
            "attributeName":"Id"
         }
      }
   ]
}

Datatype formats

In general, data producers should leverage datatype formats that aren't culture sensitive, such as ISO 8601 for datetime/datetimeoffsets. However, if a data producer writes data with culture-specific formats, the "culture" property must be set in the model.json format, and consumers must leverage that value to properly parse the data in the partition files.

Non-specified culture

When "culture" isn't specified in the model.json, consumers can assume these data formats:

Model-file attributes

We also suggest setting metadata as file properties directly on model.json for effective discovery of data-producer metadata by applications.