How can I utilize an automatic Python script to exclusively update documents with new data from a Microsoft Graph API to Microsoft Fabric Lakehouse?

Ares Inglada Graño 0

Hello everyone,

I'm seeking some assistance regarding a Python script I have uploaded in a notebook within Microsoft Fabric. Currently, the script operates automatically, facilitating data transfer from the Microsoft Graph API to the Lakehouse.

Presently, the script uploads the entire CSV file using the following code snippet:

if "access_token" in result:
    groups = getMicrosoftGraphApiGroups()

	groups.to_csv("abfss://...my route.../groups.csv", sep = "#")

However, I'm interested in modifying the script's functionality. Specifically, I'm aiming for the script to connect with the existing data and only upload new data or data from that specific day. This way, the data refreshes daily with any new information.

I would greatly appreciate any insights, guidance, or suggestions you could offer to help me achieve this goal.

Thank you in advance for your time and assistance.

Best regards,

Ares

Prrudram-MSFT 22,941 Reputation points

2023-08-31T05:08:32.8466667+00:00
@Ares Inglada Graño

To achieve your goal of only uploading new data or data from a specific day, you can modify your Python script to perform the following steps:

Read the existing data from the Lakehouse into a pandas DataFrame.

Use the Microsoft Graph API to retrieve only the new or updated data since the last time the script ran. You can do this by specifying a date range in your API query or by keeping track of the last time the script ran and only retrieving data that has been updated since then.

Merge the new data with the existing data in the pandas DataFrame.

Write the updated DataFrame back to the Lakehouse.

Share via

How can I utilize an automatic Python script to exclusively update documents with new data from a Microsoft Graph API to Microsoft Fabric Lakehouse?