RestAPI to Datalake storage

Nimai Ahluwalia 41 Reputation points
2020-11-05T13:16:13.343+00:00

Hello guys,

I am trying to pull data from the open sample API

http://dummy.restapiexample.com/api/v1/employees

And trying to load the data form it to Data Lake storage Account, but when the pipeline is debugged successfully, and when I check the Storage for the file I get only 1 row but not the whole data.

Where am I going wrong?

37743-capture.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,832 questions
0 comments No comments
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,091 Reputation points
    2020-11-05T19:47:31.847+00:00

    Hello @Nimai Ahluwalia and welcome to Microsoft Q&A.

    I have been able to reproduce , or something very close to your situation. I believe the cause is improper mapping. Using the link you shared, I tried writing to a delimited text without mapping.
    The results were

    37726-image.png
    and then when I looked at the written output I saw
    37813-image.png

    The output contains status: success
    and message: Sucessfully! all records has been fetched.

    These appear in the input, which I have abbreviated below.

    [  
      {  
        "status": "success",  
        "data": [  
          {  
            "id": 1,  
            "employee_name": "Tiger Nixon",  
            "employee_salary": 320800,  
            "employee_age": 61,  
            "profile_image": ""  
          },  
          ...many records...  
          {  
            "id": 24,  
            "employee_name": "Doris Wilder",  
            "employee_salary": 85600,  
            "employee_age": 23,  
            "profile_image": ""  
          }  
        ],  
        "message": "Successfully! All records has been fetched."  
      }  
    ]  
    

    Notice that the desired data is kept inside an inner structure, but what was written, were the fields in the outer structure. This can be solved by mapping like shown below.

    37814-image.png

    The key piece here is the "Collection reference". Setting Collection reference to the response member which contains the collecion of data you want, makes it treat each child item as a record.

    Now when I debug, I get:
    37590-image.png
    Do not worry about the discrepancy in bytes read vs bytes written. This is a result of all the extra characters in JSON notation ({ : } ) not being needed in delimited text format. You can see the data is now written.

    37824-image.png

    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.