How to Set Up Automatic D365 Finance and Operations Data Replication to Azure Data Lake for Reporting?

Nitin Patil 20

Hi everyone,

We have a D365 Finance and Operations environment hosted in the Central US region. Our Accounts Receivable (AR) data, including customers and invoices, is integrated from an on-premises Oracle database into D365 FinOps.

I would like to automatically replicate the D365 FinOps data to an Azure Data Lake. The goal is to run analytical reports and data integration quality reports using this data.

I’m following the guidance from this Microsoft documentation: Azure Synapse Link for D365 Finance and Operations.

However, I’m struggling to find a clear step-by-step process. Specifically, I need help understanding:

What specific access and permissions are required in D365 FinOps and Azure to set this up?
What roles or resources should I request from the relevant departments to proceed?

Any detailed guidance or resources would be greatly appreciated.

Thank you!

phemanth 10,240 Reputation points Microsoft Vendor

2024-09-05T10:03:50.7733333+00:00
@Nitin Patil

Welcome to the Microsoft Q&A and thank you for posting your questions here

Enable the Export to Data Lake Feature:

Go to System Administration > Setup > Export to Data Lake in your D365 FinOps environment.

If the feature isn’t available, add &mi=DataFeedsDefinitionWorkspace to your environment URL to access the setup page.

Select Data to Export:

On the Choose Tables tab, select the data tables you want to stage in the data lake.

Add the selected tables and activate the data feed.

Set Up Azure Synapse Link:

Since the Export to Data Lake feature is being deprecated, it’s recommended to use Azure Synapse Link for a more integrated experience.

Follow the Synapse Link transition guide for detailed steps on setting up Synapse Link.

Access and Permissions:

System Administrator Role:

This role is essential for configuring and managing the Export to Data Lake feature.

It allows you to access the necessary setup pages and manage data feeds.

Data Management Permissions:

Ensure you have permissions to access and manage data entities within D365 FinOps.

This includes the ability to select and activate data tables for export.

Azure:

Azure Subscription Owner or Contributor:

You need permissions to create and manage resources in Azure Data Lake Storage and Azure Synapse Analytics.

The Contributor role is typically sufficient for setting up and managing these resources.

Storage Blob Data Contributor:

This role is required to manage data within Azure Data Lake Storage.

It allows you to read, write, and delete data in the storage account.

Synapse Administrator or Synapse Contributor:

If you’re using Azure Synapse Link, you’ll need permissions to manage Synapse workspaces and linked services.

The Synapse Contributor role is usually adequate for these tasks.

Requesting Roles and Resources:

From IT or Admin Departments:

Request the System Administrator role in D365 FinOps.

In Azure, request the Contributor role for the relevant subscription and the Storage Blob Data Contributor role for the Data Lake Storage account.

If using Synapse Link, request the Synapse Contributor role for the Synapse workspace.

For more detailed guidance, you can refer to the official Microsoft documentation on exporting data to Azure Data Lake.
phemanth 10,240 Reputation points Microsoft Vendor

2024-09-09T11:28:54.5866667+00:00
@Nitin Patil

Here’s a detailed guide to help you transition to using Azure Synapse Link and connect your environment to Dataverse:

Understanding the Deprecation

The “Export to Data Lake” feature has been deprecated as of October 15, 2023, and will be available until November 1, 2024. Microsoft recommends transitioning to Azure Synapse Link, which offers enhanced performance and flexibility.

Connecting to Dataverse

To use Azure Synapse Link, you need to connect your Finance and Operations environment to a Dataverse instance. Here are the steps:

Prerequisites

Ensure you have at least 1 GB of Microsoft Power Platform database storage capacity. Validate the governance policy of your tenant in the Power Platform admin center. You need either the Global administrator role or the Power Platform administrator role.

Steps to Connect

Select Setup on the Power Platform Integration Tab:

Go to the Power Platform Integration tab in your Finance and Operations environment.

Select “Setup.”

Configure Dataverse Using a Template:

Follow the prompts to configure Dataverse using the provided template.

Confirm that you want to proceed with the setup.

Wait for Provisioning:

The provisioning process will take some time. Once completed, your environment will be connected to Dataverse.

Setting Up Azure Synapse Link

Once your environment is connected to Dataverse, you can set up Azure Synapse Link:

Access Azure Synapse Link:

Navigate to the Azure Synapse Link setup page in your Power Apps environment.

Select Finance and Operations Data:

Choose the tables and entities you want to export to Azure Synapse Analytics and Azure Data Lake Storage Gen2.

Continuous replication of entity and table data is supported.

Configure Data Export:

Configure the data export settings, including saving data in Parquet Delta Lake format for faster query response times.

here are the detailed guides you need:

Transition Guidance

Connect Finance and Operations to Dataverse

Azure Synapse Link Setup
phemanth 10,240 Reputation points Microsoft Vendor

2024-09-10T14:00:36.0933333+00:00

@Nitin Patil We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Nitin Patil 20 Reputation points

2024-09-10T14:37:33.73+00:00

@phemanth

Sorry for the delayed response—I should have provided an update yesterday. I haven't had a chance to try it yet, as I'll need Power Platform Administrator permissions to proceed. I'm still waiting on my networking team to grant those permissions.
Nitin Patil 20 Reputation points

2024-09-10T14:40:11.2366667+00:00

@phemanth I'll have an update by 09/12/2024.
phemanth 10,240 Reputation points Microsoft Vendor

2024-09-12T12:59:27.0866667+00:00

@Nitin Patil We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 10,240 Reputation points Microsoft Vendor

2024-09-13T15:26:09.21+00:00

@Nitin Patil just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Nitin Patil 20 Reputation points

2024-09-13T16:15:49.24+00:00
@phemanth

The implementation paused briefly on Wednesday (09/11/2024) due to several questions from our executive team regarding the cost, security, and irreversibility of enabling the Dataverse feature. I needed to address these concerns before moving forward. My plan is to begin by enabling the feature with just two tables in the Tier 2 sandbox environment to evaluate its impact, and then progressively add more tables as we go.

Below are the details I shared with the executives, and we will be having a meeting today (09/13/2024) to discuss these points. Once all questions are resolved, I would like to proceed with the implementation. Please let me know if you'd like any additions or updates to the information provided.

Cost Analysis

Cost of Enabling Dataverse for D365 FinOps: The cost of enabling Dataverse integration is generally included in the licensing cost of Dynamics 365 apps. Purchasing a license for these finance and operations apps includes an additional 10 GB of Dataverse database capacity. Each user license also provides incremental database capacity.

Power Platform Environments: With each license purchase, you are entitled to one self-service sandbox environment and one production environment in Lifecycle Services. Initially, the Power Platform environment created does not include a Dataverse database and uses only 1 GB of database capacity. Once Power Platform integration is set up, a Dataverse database is added, consuming 3 GB or more of capacity.

Add-On Environments and Capacity: Additional sandbox environments purchased for Lifecycle Services do not come with extra Dataverse database capacity. For additional Power Platform integration needs, customers must purchase add-on Dataverse storage. Microsoft provides additional storage in increments, typically costing $40 per GB per month. (For our scenario, it is anticipated that we will not need additional Dataverse storage, as the estimated requirement for customer and invoice-related tables is less than 10 GB. Actual storage consumption will be confirmed post-migration. For precise pricing, consult Microsoft’s official pricing details or contact a Microsoft sales representative.)

Power Platform Costs: Registering Microsoft.PowerPlatform in the DEV subscription can be done at no additional cost using the Power Apps Developer Plan, which offers a free development environment. The Power Apps premium plan costs approximately $20 per user per month. (For our scenario, development incurs no cost, but production could cost $20 per month per user plus any additional charges for Dataverse storage if needed.)

ADLS2 Storage Costs: For Azure Data Lake Storage (ADLS) Gen2, the cost varies by storage tier:

Hot Tier: $0.021 per GB per month

Cool Tier: $0.016 per GB per month

Cold Tier: $0.0036 per GB per month

Transaction costs for each tier are:

At $0.065 per 10,000 write operations: $0.65 per month

At $0.13 per 10,000 write operations: $1.30 per month

At $0.234 per 10,000 write operations: $2.34 per month

Additional costs include read operations, geo-replication, outbound data transfer, and changing the default account access tier_. (Initially, we expect to remain well below $XX per month. Costs may increase as data volumes and usage patterns change. Regular reviews and updates to cost calculations are recommended.)_

Azure Synapse Analytics Costs: Querying data from ADLS2 using Azure Synapse Analytics costs $5 per TB of data processed.

2. Data Security

Data access will be secured using Role-Based Access Control (RBAC. Azure Synapse supports encryption in transit and at rest. As an Azure Data Solution Architect, I will ensure:

Encryption settings are enabled for data at rest and in transit

Access control and Azure roles are properly implemented

Row-Level and Column-Level Security are established

Dynamic data masking and conditional access are enabled

Data exfiltration protection is implemented to monitor and control data movement out of the Synapse workspace

May use ACL in parallel with RBAC

3. Irreversibility of Power Platform Connection

From my research, "irreversible" in the context of linking a Finance and Operations apps environment to a Microsoft Dataverse instance means that once the integration is established, it cannot be undone without significant consequences. Disconnecting the systems may cause inconsistencies: while the D365 data remains intact, the Dataverse environment will be impacted. Thus, "irreversible" indicates that reversing the integration results in permanent effects on Dataverse, though the D365 data remains unaffected.

4. Tier 1 Cost

I did not pursue this further as configuring the environment to work with Dataverse would require additional effort and potentially involve extra settings. However, it’s important to note that deleting the Dataverse environment will not affect the D365 data.
Nitin Patil 20 Reputation points

2024-09-13T16:29:00.9+00:00

@phemanth The implementation paused briefly on Wednesday due to several questions from our executive team regarding the cost, security, and irreversibility of enabling the Dataverse feature. I needed to address these concerns before moving forward.

1 answer

Nitin Patil 20 Reputation points

2024-09-05T16:01:11.37+00:00

@phemanth

Thank you for your prompt response.

I am a System Administrator in D365 Finance and Operations and a contributor to our Azure subscription.

I’ve encountered an issue with the "Export to Data Lake" feature within D365 Finance and Operations. Upon launching the Export to Data Lake page, I receive a message stating that the feature is temporarily unavailable and to check back later. This message remains even after selecting a table (e.g., Customers - CustTable).

Additionally, under the Action Center, I see a message suggesting that I connect this Finance and Operations environment to a Dataverse instance. Is it necessary to connect to Dataverse when using the Synapse Link? If so, where can I find detailed guidance on how to accomplish this?

I've reviewed the transition guidance for the Export to Data Lake feature at the following link:

Transition Guidance

However, it doesn’t provide the step-by-step instructions I need to complete this setup.

I also attempted to follow the instructions in this link:

Azure Synapse Link Setup

But it lacks the specific steps I need to take.

Could you please provide any specific guidance?

Thanks,

Nitin
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to Set Up Automatic D365 Finance and Operations Data Replication to Azure Data Lake for Reporting?

1 answer

Your answer