Spark job definition with blob fileName and folderPath is not working

Just to clarify, I am using Azure Synapse studio for building the data pipeline

@Johny Yagappan

Thanks for using MS Q&A platform and posting your query.

It seems like the issue is with the way you’re passing the parameters to your Spark job. The @triggerBody().fileName and @triggerBody().folderPath are expressions that get evaluated in the context of the pipeline, not in the Spark job. When you pass these expressions as command line arguments, they are passed as is, without being evaluated, hence you see @triggerBody().fileName @triggerBody().folderPath as the output of sys.argv[1:].

To resolve this issue, you need to ensure that the parameters are evaluated before they are passed to the Spark job. Here’s how you can do it:

1. In your pipeline, create two new parameters, let’s call them resolvedFileName and resolvedFolderPath.
2. Set the default values of these parameters to @triggerBody().fileName and @triggerBody().folderPath respectively. This will ensure that the expressions are evaluated in the context of the pipeline.
3. In your Spark job definition, replace pipeline().parameters.fileName and pipeline().parameters.folderPath with pipeline().parameters.resolvedFileName and pipeline().parameters.resolvedFolderPath respectively.

Here’s how the updated command line arguments expression would look like:

@array(concat(pipeline().parameters.resolvedFileName,' ',pipeline().parameters.resolvedFolderPath))

Now, when you trigger the pipeline, the resolvedFileName and resolvedFolderPath parameters should contain the actual file name and folder path, which will then be passed to your Spark job as command line arguments.

Hope this helps. Do let us know if you any further queries.

@Johny Yagappan We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thanks Phemanth for the response. This does not work for me. I figured that issue and fixed it. The issue was with the trigger definition where these parameters have no value set. when I set the default value as follow fileName to @triggerBody().fileName and folderPath to @triggerBody().folderPath. Then I was able to access the fileName and folderPath values in the pyspark script.

Thanks for your response. Much appreciated.