Encryption Data on transit - from oracle to Azure Data lake

Xhevahir Mehalla 160 Reputation points
2024-07-02T07:19:25.15+00:00

Hello -

We have this implementation setup:

  1. use Synapse analytics Pipelines to extract data from Oracle OCI database
  2. Copy data to Azure data Lake Storage
  3. Transform data from ADLS Gen 2 to Azure Sql Database

We want to make sure that :

  1. Data extracted from Oracle OCI is encrypted/secure before you load them to ADLS gen 2 (what do we use to do that - Encryption mechanism). This will be data on Transit I guess
  2. While we extract data using Synapse Pipelines; are there any features to consider to secure/encrypt data?
  3. We know that in ADLS Gen 2 the data is encrypted or can be encrypted with a key. This is secure data at rest.

Please can someone share some lights how to secure data on Transit and at rest while copying/extracting from Oracle OCI to ADLS gen 2.

Thank you

1.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,409 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,621 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ramya Harinarthini_MSFT 5,316 Reputation points Microsoft Employee
    2024-07-02T08:04:46.35+00:00

    @Xhevahir Mehalla Welcome to Microsoft Q&A Forum, Thanks for posting here!!

    Ensuring the security of data both in transit and at rest during the process of extracting from Oracle OCI, storing in Azure Data Lake Storage (ADLS) Gen 2, and transforming to Azure SQL Database involves several key steps and features. Here's a detailed look at how you can secure your data at each stage:

    1. Securing Data in Transit

    To secure data in transit while extracting from Oracle OCI and loading it to ADLS Gen 2, you can use the following encryption mechanisms and features:

    a. Transport Layer Security (TLS)

    • Oracle OCI: Ensure that the connection to the Oracle OCI database uses TLS (Transport Layer Security). This ensures that the data being transferred is encrypted. Oracle supports SSL/TLS for secure data transfer.
    • Synapse Pipelines: When setting up linked services in Synapse Pipelines to connect to Oracle OCI, make sure to configure the connection to use TLS/SSL.

    b. Managed Virtual Network

    • Azure Synapse offers a managed virtual network that you can enable to secure communication between your services. This helps to keep your data transfer within a private network, reducing the risk of interception.
    1. Securing Data at Rest

    a. Azure Data Lake Storage Gen 2

    • Encryption at Rest: Data in ADLS Gen 2 is encrypted at rest by default using Microsoft-managed keys. You also have the option to use customer-managed keys (CMKs) for encryption.
    • Azure Key Vault: For managing your encryption keys, Azure Key Vault can be used. This allows you to maintain control over the keys used to encrypt your data.

    b. Azure Synapse Analytics and SQL Database

    • Data Encryption: When transforming and loading data into Azure SQL Database, ensure that Transparent Data Encryption (TDE) is enabled. TDE provides encryption of data at rest for the entire database.
    • Always Encrypted: For sensitive data, consider using Always Encrypted, which ensures that sensitive data within a database is encrypted both at rest and in use.
    1. Features in Synapse Pipelines for Security

    Synapse Pipelines offer several features to enhance security:

    a. Data Movement Security

    • Integration Runtimes: Use Azure Integration Runtime or a Self-hosted Integration Runtime (SHIR) to securely move data. Integration Runtimes can be configured to use secure credentials and TLS for data transfer.
    • Linked Services: Ensure that linked services are configured to use secure authentication methods, such as Managed Identity or service principal with strong passwords or certificates.

    b. Access Control

    • Role-Based Access Control (RBAC): Implement RBAC to control who has access to your Synapse workspace and pipelines. Assign appropriate roles to users to ensure that only authorized personnel can access or modify your data flows.

    Best Practices

    1. Enable Encryption: Ensure that encryption is enabled at every stage of your data flow - both in transit and at rest.
    2. Use Secure Protocols: Always use secure protocols (such as TLS) for data transfer.
    3. Key Management: Use Azure Key Vault for managing your encryption keys and ensuring that they are rotated regularly.
    4. Audit and Monitoring: Enable logging and monitoring to track access and changes to your data. Azure Monitor and Azure Security Center can help you with continuous monitoring and alerting.

    By following these guidelines and leveraging the built-in security features of Azure Synapse Analytics, ADLS Gen 2, and Azure SQL Database, you can ensure that your data is secure both in transit and at rest during the entire ETL process

    Hope this helps!

    Kindly let us know if the above helps or you need further assistance on this issue

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members

    0 comments