Databricks connectivity to Eventhubs error

Question

Hi
I am connecting from Azure Databricks to event hubs using python program. I used the program from microsoft documentation. Python program runs well and inserts events into eventhub when run locally from my pc. When I run the program from databricks cluster, it fails with following error:

import time
import os
import uuid
import datetime
import random
import json
import asyncio
from azure.eventhub import EventData, EventHubProducerClient

producer = EventHubProducerClient.from_connection_string(conn_str="Endpoint=sb://eventhubnamespace.servicebus.windows.net/;SharedAccessKeyName=saspolicyname;SharedAccessKey=saspolicykey", eventhub_name="ehname")

event_data_batch = producer.create_batch()

AuthenticationError: Unable to open authentication session on connection b'EHProducer-20ebba19-ffd3-48af-827e-d2043372a71d'.

I verified that sas access name and key are fine. Similary, I verified eventhub name is also correct. The same statement works perfectly well in stand alone python program. It only happens when I run this on databricks cluster. On doing some research , I found that some solutions mentioned to open port 5671 and 5672. We also opened these ports. Am I missing anything? please advice.

Thanks

Accepted Answer

Hi
Thanks for the response. I had the correct target hostname. For who ever is facing this issue, this solution may be helpful in the future. Issue was with port #. Python SDK in Databricks by default is trying to connect to port 5671 or5672 using AMQP. Python standalone program ".py" on my local system is by default is connecting to 443 port. This issue was resolved by using additional parameter: transport_type. By default transport_type connects to 5671 or 5672 port. To override it to connect to 443, I used transport_type=TransportType.AmqpOverWebsocket

producer = EventHubProducerClient.from_connection_string(conn_str="Endpoint=sb://eventhubnamespace.servicebus.windows.net/;SharedAccessKeyName=saspolicyname;SharedAccessKey=saspolicykey", eventhub_name="ehname", transport_type=TransportType.AmqpOverWebsocket)

This following link has the details on port and firewall:

https://video2.skills-academy.com/en-us/azure/event-hubs/event-hubs-faq

https://video2.skills-academy.com/en-us/python/api/azure-eventhub/azure.eventhub.aio.eventhubproducerclient?view=azure-python

Answer

Hello @kranthi k senapathi ,

Welcome to the Microsoft Q&A platform.

You will experience this error AuthenticationError: Unable to open authentication session on connection b'EHProducer-20ebba19-ffd3-48af-827e-d2043372a71d'. message due to incorrect target hostname passed in the EventHub connection string field.

Note: Make to get the correct connection string from the portal.

Sign in to Azure portal.
Select All services on the left navigational menu.
Select Event Hubs in the Analytics section.
In the list of event hubs, select your event hub.
On the Event Hubs Namespace page, select Shared Access Policies on the left menu.
Select a shared access policy in the list of policies. The default one is named: RootManageSharedAccessPolicy. You can add a policy with appropriate permissions (read, write), and use that policy.
Select the copy button next to the Connection string-primary key field.

I'm able to execute above query by passing the correct connection string of the Azure Event Hub.

For more details, refer to Send events to or receive events from event hubs by using Python (azure-eventhub).

Hope this helps. Do let us know if you any further queries.

------------

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Share via

Databricks connectivity to Eventhubs error

1 additional answer

Your answer