Databricks not sending audit logs to event hub

Danny AMAYA 0 Reputation points
2024-06-13T19:41:27.85+00:00

Hi, I'm trying to push all the logs from databricks using the diagnostic tool to event hub but is not working, it didn't push anything.

I'm using the root access policy and also already created the eventhub name, what else I'm probably missing?

Thanks in advance.

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
585 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,042 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. hossein jalilian 4,690 Reputation points
    2024-06-13T23:26:42.6066667+00:00

    Thanks for posting your question in the Microsoft Q&A forum.

    Here are some steps you can take to troubleshoot why Databricks might not be sending audit logs to the Event Hub:

    • Ensure the Event Hub namespace and the specific Event Hub instance you are using exist and are accessible. check that the shared access policy you are using has the necessary permissions to send data to the Event Hub.
    • Go to the Databricks workspace, navigate to Manage > Audit Logs > Diagnostic Settings. Verify that the diagnostic setting is enabled and the correct Event Hub is selected as the destination. Check if the RuntimeAuditLogs category is selected to be sent to the Event Hub.
    • Check the Event Hub metrics in the Azure portal to see if any data is being received.
    • In the diagnostic settings, click on the Test connection button for the Event Hub destination. This will verify if Databricks can successfully connect and send data to the specified Event Hub.

    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful

  2. PRADEEPCHEEKATLA-MSFT 83,966 Reputation points Microsoft Employee
    2024-06-18T04:15:25.49+00:00

    @Danny AMAYA - Thanks for the question and using MS Q&A platform.

    It seems like you are having trouble pushing logs from Azure Databricks to Event Hub using the diagnostic tool. Can you please provide more information about the steps you have taken so far?

    Also, have you checked if the diagnostic settings for Azure Databricks Workspace to stream resource logs to a Log Analytics workspace is enabled? This is a prerequisite for streaming logs to Event Hub. You can find more information about this in the Azure documentation: Configure diagnostic log delivery

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.