Scenario - Integrate Human Resources data with an external system
Data integration can be asynchronous, and complex data transformation and filtering is required.
Pattern - Synchronize data to Azure Data Lake Storage for integration
Consider the following patterns (best practices):
- Set up the required tables and entities in the Export to Azure Data Lake feature.
- Use a SQL pool in Microsoft Azure Synapse Analytics for storage and transformation.
- Build integrations from Azure Synapse Analytics where data is replicated.
- Perform further transformation and filtering through Azure Data Factory, Azure Synapse SQL stored procedures, and so on.
Further information
Using the Azure Data Lake Storage export feature is the best and most efficient way to extract large volumes of data from the application in a read-only mode.
Azure Synapse also provides you with the ability to store, organize, and manipulate the data in multiple stages. As a result, the transformation and filtering process are easier and more streamlined than any custom approach.
Anti-patterns
Consider the following anti-patterns (pitfalls):
- Synchronize all entities to the Bring your own device (BYOD) from Human Resources.
- Use OData integration patterns.
- Create custom services integration patterns.