Databricks Cluster Logs Driver stdout - delayed in dbfs location

Ayushri Jain 181 Reputation points
2021-05-25T06:50:00.98+00:00

I have a databricks cluster with logging enabled in a dbfs location. My process requires to read the cluster logs, specifically the driver/stdout logs. This stdout is nothing but the console output which is also visible in UI : Clusters -> ClusterName -> Driver Logs -> Standard Output.
Every time, the UI shows me the stdout information very quickly but when I try to read the same from dbfs location, it's not yet present there and usually takes 4-5 minutes to reach there.
Is this delay expected? Is there any way to immediately sync the UI stdout and the dbfs stdout logs?

Enterprise Databricks configuration - 7.5 ML (includes Apache Spark 3.0.1, Scala 2.12)

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 90,226 Reputation points Microsoft Employee
    2021-05-27T10:03:31.287+00:00

    Hello @Ayushri Jain ,

    According to the document - Logs are delivered every five minutes to your chosen destination.

    It’s an expected behavior. I don't believe there is a way to circumvent the process.

    Hope this helps. Do let us know if you any further queries.

    ---------------------------------------------------------------------------

    Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.