Configure REST in a copy activity
This article outlines how to use the copy activity in data pipeline to copy data from and to a REST endpoint.
Specifically, this generic REST connector supports:
- Copying data from a REST endpoint by using the GET or POST methods and copying data to a REST endpoint by using the POST, PUT or PATCH methods.
- Pagination in the REST APIs.
- For REST as a source, copy the REST JSON response as-is or parse it by using schema mapping. Only the response payload in JSON is supported.
Supported configuration
For the configuration of each tab under a copy activity, go to the following sections respectively.
General
For General tab configuration, go to General.
Source
The following properties are supported for REST under the Source tab of a copy activity.
The following first three properties are required:
- Data store type: Select External.
- Connection: Select an REST connection from the connection list. If no connection exists, then create a new REST connection by selecting New.
- Connection type: Select REST.
- Relative URL: A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the connection definition is used. The HTTP connector copies data from the combined URL:
[URL specified in connection]/[relative URL specified].
Under Advanced, you can specify the following fields:
Request method: The HTTP method. Allowed values are GET (default) and POST.
Request timeout: The timeout (the timespan value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is 00:01:40.
Request interval (ms): The interval time between different requests for multiple pages in milliseconds. Request interval value should be a number between [10, 60000].
Additional headers: Additional HTTP request headers.
Pagination rules: The pagination rules to compose next page requests. Refer to the pagination support section for details.
Additional columns: Add more data columns to store source files' relative path or static value. If you're using a static value, you can use custom expressions to generate it.
Destination
The following properties are supported for REST under the Destination tab of a copy activity.
The following first three properties are required:
- Data store type: Select External.
- Connection: Select a REST connection from the connection list. If no connection exists, then create a new REST connection by selecting New.
- Connection type: Select REST.
- Relative URL: A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the connection definition is used. The HTTP connector copies data from the combined URL:
[URL specified in connection]/[relative URL specified].
Under Advanced, you can specify the following fields:
Request method: The HTTP method. Allowed values are POST (default), PUT, and PATCH.
Request timeout: The timeout (the timespan value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to write the data. The default value is 00:01:40.
Request interval (ms): The interval time between different requests for multiple pages in milliseconds. Request interval value should be a number between [10, 60000].
Write batch size: Number of records to write to the REST destination per batch. The default value is 10000.
Http Compression type: HTTP compression type to use while sending data with Optimal Compression Level. Allowed values are None and GZip.
Additional headers: Additional HTTP request headers.
Mapping
For the Mapping tab configuration, go to Configure your mappings under mapping tab. Mapping is not supported when both source and destination are hierarchical data.
Settings
For the Settings tab configuration, go to Configure your other settings under settings tab.
Table summary
The following tables contain more information about the copy activity in REST.
Source information
| Name | Description | Value | Required | JSON script property |
|---|---|---|---|---|
| Data store type | Your data store type. | • Workspace • External • Sample dataset |
Yes | type |
| Connection | Your connection to the source data store. | <your connection> | Yes | connection |
| Connection type | Your connection type. Select REST. | REST | Yes | type: RestResource |
| Relative URL | A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the connection definition is used. The HTTP connector copies data from the combined URL: [URL specified in connection]/[relative URL specified]. |
<your relative url> | No | relativeUrl |
| Request method | The HTTP method. Allowed values are GET (default) and POST. | • GET • POST |
No | requestMethod |
| Request timeout | The timeout (the timespan value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is 00:01:40. | timespan | No | httpRequestTimeout |
| Request interval (ms) | The interval time between different requests for multiple pages in milliseconds. Request interval value should be a number between [10, 60000]. | [10, 60000] | No | requestInterval |
| Additional headers | Additional HTTP request headers. | <your additional headers> | No | additionalHeaders |
| Pagination rules | The pagination rules to compose next page requests. Refer to the pagination support section for details. | Go to pagination | No | paginationRules |
| Additional columns | Add more data columns to the store source files' relative path or static value. Expression is supported for the latter. | • Name • Value |
No | additionalColumns: • name • value |
Destination information
| Name | Description | Value | Required | JSON script property |
|---|---|---|---|---|
| Data store type | Your data store type. | • Workspace • External |
Yes | type |
| Connection | Your connection to the source data store. | <your connection> | Yes | connection |
| Connection type | Your connection type. Select REST. | REST | Yes | type: RestResource |
| Relative URL | A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the connection definition is used. The HTTP connector copies data from the combined URL: [URL specified in connection]/[relative URL specified]. |
<your relative url> | No | relativeUrl |
| Request method | The HTTP method. Allowed values are POST (default), PUT, and PATCH. | • POST • GET • PATCH |
No | requestMethod |
| Request timeout | The timeout (the timespan value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is 00:01:40. | timespan | No | httpRequestTimeout |
| Request interval (ms) | The interval time between different requests for multiple pages in milliseconds. Request interval value should be a number between [10, 60000]. | [10, 60000] | No | requestInterval |
| Write batch size | Number of records to write to the REST destination per batch. The default value is 10000. | <number of rows> (integer) |
No | writeBatchSize |
| Http Compression type | HTTP compression type to use while sending data with Optimal Compression Level. Allowed values are None and GZip. | • None • GZip |
No | httpCompressionType |
| Additional headers | Additional HTTP request headers. | <your additional headers> | No | additionalHeaders |
Related content
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for