Azure Batch equivalent of FILE.AppendAllText

etl2016 1 Reputation point
2020-06-26T16:01:24.083+00:00

Hi,

Could you please let me know the Azure Batch Service (Storage Account blob processing) equivalent of C# FILE.AppendAllText?

Basically, I have an input file. I need to read it line by line with Stream Reader. Do transformation. Write the output file to Blob Storage Account. And this should happen line by line in Azure

thank you

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
320 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. prmanhas-MSFT 17,901 Reputation points Microsoft Employee
    2020-07-06T11:06:10.77+00:00

    @etl2016-6749 Apologies for the delay in response.

    Azure batch service is quite different from File.AppendAllText functionality of C#. The approaches below assumes you want to process a file with several lines on multiple azure batch jobs.

    In order to partition the data to into multiple jobs, I can suggest following approaches

    1. Partition of job is handled in the application side. So application divide the file into 4 jobs and keep track of progress of all 4 jobs and then can perform the reduce/aggregation operation on all 4 jobs.
      This is easier to handle at application side but difficult to scale in case there are multiple such jobs running simultaneously.
    2. You can perform the partitioning operation in the batch service also (as custom job). This job then can spawn 4 jobs and 1 reduce/aggregation job which is scheduled after completion of 4 jobs. (Dependent jobs are supported in batch service).
      This is easier to scale but would require handling in application side as the initial job would be complete but sub-jobs are still running.

    Hope this helps.

    Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics.

    0 comments