Ingest data with Fabric link

Articolo
11/12/2024

Follow the steps in this article to bring your organization’s environmental data from the Microsoft Sustainability Manager data model in Dataverse to the IngestedRawData lakehouse in your Fabric workspace by using a Fabric link.

Prerequisites

Deploy the Environmental data and insights capability in Fabric workspace.
Ensure that Sustainability Manager is installed and set up in your Dataverse environment.
Ensure you have the Dataverse system administrator security role.
Ensure you're an administrator of the Fabric workspace.
Enable Link to Fabric from the Dataverse environment.

Step 1: Set up link to Microsoft Fabric

In this step, you set up a link to Microsoft Fabric between the Sustainability Manager environment from which you want to bring your data and the Sustainability data solutions in Microsoft Fabric workspace.

Open the Power Apps portal. On the top right, select your Dataverse environment where Sustainability Manager is installed.
Select Tables from the left pane and select Analyze, and then select Link to Microsoft Fabric on the command bar.

A wizard opens that validates your Fabric subscription and connection settings. You can set up and save a one-time connection to enable Fabric to securely access data from your Dataverse environment. Select Next to proceed.

The wizard asks you to select an existing Fabric workspace or to create a new one. Here, select the Fabric workspace where you deployed Sustainability data solutions in Fabric.

Select Create to proceed. You can see the shortcuts to all your tables in a new lakehouse in the selected workspace.

Step 2: Create a Microsoft OneLake shortcut

In this step, you create shortcuts in the IngestedRawData lakehouse for the Sustainability Manager tables that you want to transform into the environmental, social, and governance (ESG) data model.

Open the workspace in Fabric where you deployed the Sustainability solution. To open the solution, select it from the list view. Select the ESG data estate capability from the Manage deployed capabilities list view to reach the managed capability page.
From the ESG data estate managed capability page, select the IngestedRawData lakehouse.

From the lakehouse, choose Get data, and then select New shortcut.

Select Microsoft OneLake as the shortcut type.

From the list of lakehouses, select the newly created lakehouse with shortcuts to all Sustainability Manager tables (created in Step 1) as the data source type. Select Next to proceed.

Choose the Sustainability Manager tables that you want to transform, and then select Next. These table names usually start with the prefix msdyn_.

The tables are now available in the IngestedRawData lakehouse and are ready to be transformed into the ESG data model.

Step 3: Transform data

After the Sustainability Manager data integrates into the IngestedRawData lakehouse, unify and harmonize the data by transforming it into the ESG data model schema. During deployment, the ProcessedESGData lakehouse was created in the workspace for storing the harmonized data. The ESG data model schema links the ESG data entities to other data entities that describe other functional areas of a company, such as HR, IT, or finance production.

For transforming Sustainability Manager data to the ESG data model schema, the capability provides notebooks and data pipelines that use prebuilt transformation and orchestration technologies using libraries stored in the ConfigAndDemoData lakehouse in your workspace.

Run the TransformMSMDataToProcessedESGData_DTPL data pipeline to load and transform the Sustainability Manager data to the ESG data model schema in the ProcessedESGData lakehouse.

The first step is deactivated by default and isn't required for this method, because the data from Sustainability Manager is already loaded as lakehouse tables.
The transformation of the Sustainability Manager data to the ESG data model schema is a two-stage process:

First, the TransformMSMDataToProcessedESGData_INTB notebook transforms the Microsoft Sustainability Manager data to the intermediary raw data import tables.
Then, the TransformRawImportESGDataToProcessedESGData_INTB notebook transforms the data from raw data import tables to the ESG data model schema.