How to copydata from RestApi using datafactory and save it in Datalake?

stramzik 161 Reputation points
2020-07-27T11:18:38.973+00:00

Hi Everyone,

I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here
https://video2.skills-academy.com/en-us/azure/data-factory/connector-rest & https://www.alexvolok.com/2019/adfv2-rest-api-part1-oauth2/

The API which I'm trying to connect uses OAuth2 so I need to first get the access token and then do a get request to get actual data.

Below are the steps which I'm following

  1. Creating a Web HTTP request in the pipeline and passing the client_ID, client secret, username, password and grant type in the body of the request. When I debug the pipline I do get the Access_token which I need in step 2.
  2. In Step two I have a copy activity which uses the output(access_token) from web to authenticate the second REST GET request but this is where I'm facing a lot of issues. The code which I'm using is "@markus.bohland@hotmail.de ('Bearer ', activity('GetAccessToken').output.access_token)"
  3. In step 3 I have two datasets and 2 Linked services, Dataset 1 is a REST dataset which has the base url and relative url which is linked to the REST linked service and secondly the sink dataset is connected to AZURE datalake storage.
  4. In Source Dataset I'm passing additional header Authorization = @markus.bohland@hotmail.de ('Bearer ', activity('GetAccessToken').output.access_token) and ideally since the API which I want to call will return empty if no parameters are send so I pass in the parameters inside the "Request Body" is that even correct? the Request body would look something like this "start_date=2020/07/17&end_date=2020/07/18".
  5. The sink is a simple Json dataset stored in DataLake.

When I try to debug I get the error as below

But I'm getting the below error
{ "errorCode": "2200", "message": "Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 401 indicates failure. { \"Error\": { \"Message\":\"Authentication failed: Invalid headers\", \"Server-Time\":\"2020-07-27T06:59:24\", \"Id\":\"6AAF87BC-5634-4C28-8626-810A19B86BFF\" } },Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "CopyDataFromAPI", "details": [] }

Please advise if I'm doing anything wrong.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,518 questions
{count} votes

Accepted answer
  1. stramzik 161 Reputation points
    2020-07-29T08:07:02.68+00:00

    I Knew this was a simple issue,

    So for people who are looking for answers.

    Please make sure the REST Source URL starts with HTTPS:// instead of HTTP:// I Guess Azure does not pass headers to url which starts with HTTP://

    4 people found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.