Azure Data Factory - Rest API call to HubSpot- Copy Activity - Response provide URL but missing hapikey, How to append hapikey & paginate to get all records

Vikas Brahmbhatt 1 Reputation point
Jul 14, 2022, 2:34 PM

Hello,

I know similar questions are asked but somehow I have tried many of that options and did not succeed. Need help with a clear step-by-step guide on how to resolve this.

I am using the ADF Copy activity to fetch deal data from HubSpot. Following are the screenshots of my current setup.

Linked Service:
Authentication Type: Anonymouse

Base URL: https://api.hubapi.com/crm/v3/objects/deals?limit=100&properties=dealname%2Cdealtype%2Cpipeline%2Cregion__c %2Ccreatedate&hapikey=<<key>>

220841-image.png

Dataset:
Base URL: https://api.hubapi.com/crm/v3/objects/deals?limit=100&properties=dealname%2Cdealtype%2Cpipeline%2Cregion__c %2Ccreatedate&hapikey=<<key>>
Relative URL:
220787-image.png

Copy Activity: Source Setting are as follows
220814-image.png

HubSpot API is returning the next page URL but it does not have hapikey.

Using the above method how do I append the hapikey and iterate to read all the deals data ?

HubSpot API does not return a count of pages that need iteration so can not use For Loop.
Many have suggested in past to use OAuth2 client credential instead of hapikey but not sure how to setup it up as not sure what will be the token endpoint?

Any suggestions on how to append hapikey to the next page URL and make copy activity iterate will help.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,961 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.