File(filePath).exists does not work in Azure databricks

Rajaniesh Kaushikk 476 Reputation points
2020-06-11T20:12:45.693+00:00

Hi,

How to find if file exists in a path in the data lake?

Regards
Rajaniesh

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 84,381 Reputation points Microsoft Employee
    2020-06-12T09:28:04.56+00:00

    @Rajaniesh, Welcome to the Microsoft Q&A platform.

    From the error message, java.lang.Exception: Could not find file xxx /mnt/raw/File.json, looks like you are passing incorrect path.

    Make sure the path exists and also try the steps mentioned below and do let us know the status.

    9941-adb-filepath-exists.jpg

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    0 comments No comments

  2. Rajaniesh Kaushikk 476 Reputation points
    2020-06-14T04:01:39.583+00:00

    Thanks Pradeep but this is the code used to read the file with spark.read which works fine for me as well. my path is also correct the but it does not works.

    So I made this change

    import java.io.File
    import scala.io.Source
    import scala.util.parsing.json.JSON

    class SchemaJsonUtil(val schemaFilePath: String) extends java.io.Serializable {

    private def loadFile(filePath: String): Map[String, List[String]] = {
    var FileLines=""
    val filename = ("/dbfs/").concat(schemaFilePath)
    for (line <- Source.fromFile(filename).getLines()) {
    FileLines=FileLines.concat(line)
    }
    val jsonString = FileLines
    val jsonMap: Map[String, Any] = JSON.parseFull(jsonString).get.asInstanceOf[Map[String, Any]]
    jsonMap.map { case(fix, typos) =>
    (fix, typos.asInstanceOf[List[String]])
    }.toMap
    }

    Still the issue remains same that how do we check if the path passed to the loadfile function is valid and file exists there or not? Since it is not working I removed the error checking code.Please let me know how do we find if the path is valid?

    Regards
    Rajaniesh

    0 comments No comments