Is it possible to encrypt and compress files on self-hosted runtime as part of ADF flow?

@Sandeep Wadhwa I do not know which cloud storage you plan to use, but some of them transparently encrypt data-at-rest.

If you still want to do encryption before compression, I think you will need to create a script to do that outside of ADF. I am not aware of any feature native to ADF for encrypting on-prem files at rest.

There are three ways I can think to integrate the encryption and flow.

1. You on-prem script encrypts the files. When ready, the script triggers the pipeline, passing the path of the file to copy.
2. Open an endpoint so ADF can use a Webhook activity to tell the script to encrypt the files. When completed, ADF copies the files.
3. If you have a database server on prem, ADF could use a stored proc to start the script to encrypt the files.

Thanks Martin - that is helpful.

A follow-up question on this - since the compression is specified on the sink, the compressed files will be copied over to Sink. Uncompressing these files at the sink, is that something that has to be handled externally or it can be done as part of ADF flow?

While decompression can be handled externally, you could also do it in ADF as a second copy activity. ( Cloud storage compressed --> cloud storage decompressed)

That makes sense. My sink is an SFTP - if I use ADF copy flow with not IR on sink, Azure compute will be used for decompression - is that right and that will unnecessary back and forth of data.
From design perspective, I think decompressing externally may make sense and it can be post-cursor to copy activity that can be invoked from within ADF using web hook.

Let me know what you think.

So your process is:
on-prem --> Azure Storage --> SFTP ?

Why not
on-prem --> SFTP ?

If neither source nor sink uses self-hosted IR, then Azure IR is used. This means the second copy activity for decompression will use Azure compute.
Storage is storage. Decompression needs a compute to happen. To minimize the movement of data during decompression, use an IR / compute in the same datacenter / region.

If you need to decrypt at the same time as decompress, doing both in one external operation makes sense, as this reduces the back and fourth. Also can reduce writes to storage. Triggering this by webhook or web activity makes sense to me.