HD Insights - converting the Aggregated Logs output into ADLSGen2 into Text format
Can someone share a reference to systematically output logs from the HDinsights application runs into a readable format -- for the stdout. Today when running a Spark application through HdInsights (triggered by Livy API), after the application finishes the logs are accessible in HTML Format at location of https://<cluster-name>.azurehdinsights.net/yarnui/jobhistory/logs/{clusteruri}/port/30050/{container}/livy.
In addition, the logs are saved in the default storage location at /app-logs/livy/logs/{appid}. However these aggregated logs are encrypted in a binary format and are unable to be read. Is there a way to save these outputs in textual format or access the response via API? This would allow for systematically adding logs to failed applications and eliminate need to manually load the HTML response to find a stack trace.