HDInsight Hive error: Unable to rename
When you use Azure PowerShell to run HiveQL statements against HDInsight or HDInsight emulator, you might get the following error:
Failed with exception Unable to rename:
wasb://xxxxxxx@xxxxxxxx.blob.core.windows.net/hive/scratch/hive_2014-02-12_14-58-27_594_640725686036024640-1/-ext-10001
to: wasb://xxxxxxxx@xxxxxxxx.blob.core.windows.net/apps/hive/warehouse/delays
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
Creating a /apps/hive/warehouse folder on HDInsight can bypass this error message, but could run into another problem. You might notice a newly created Hive table doesn't contain any records. In the error log, you could find the data file was moved to the /apps/hive/warehouse folder instead of the /hive/warehouse folder.
This problem was fixed in HDInsight clusters version 3.1 and higher, with the exception of HBase deployments (fix in the works).
**
To automate this fix where needed:
**You can use PowerShell to deploy HDInsight following this blog article and this documentation with a custom configuration for the Hadoop Core settings, to include this workaround each time you create a cluster from PowerShell scripts.
These are the two key syntax lines to use with a code sample.
# Customize Core configuration options
$CoreConfigValues = @{ "templeton.hive.properties"="hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/hive/warehouse/" }
...
| Add-AzureHDInsightConfigValues -Core $CoreConfigValues
...
This ends up putting the value in this core-site.xml file on the nodes:
"C:\apps\dist\hadoop-2.4.0.2.1.6.0-2103\etc\hadoop\core-site.xml"
<property>
<name>templeton.hive.properties</name>
<value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/hive/warehouse/</value>
</property>
To fix the problem on HDInsight cluster version 3.0:
Connect to the cluster head node using RDP. For instructions, see Administer HDInsight clusters using Management portal.
Use a text editor to open C:\apps\dist\hcatalog-0.12.0.2.0.7.0-1528\etc\webhcat\webhcat-site.xml.
Find the following tags:
<property> <name>templeton.hive.properties</name> <value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/apps/hive/warehouse</value> <description>Properties to set when running hive.</description> </property>Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.
To fix the problem on HDInsight cluster version 2.1:
Connect to the cluster head node using RDP. For instructions, see Administer HDInsight clusters using Management portal.
Use a text editor to open C:\apps\dist\hcatalog-0.11.0.1.3.2.0-05\conf\webhcat-site.xml.
Find the following tags:
<property> <name>templeton.hive.properties</name> <value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/apps/hive/warehouse</value> <description>Properties to set when running hive.</description> </property>Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.
To fix the problem on HDInsight emulator:
Use a text editor to open C:\Hadoop\hive-0.9.0\conf\hive-site.xml.
Find the following tags:
<property> <name>hive.metastore.warehouse.dir</name> <value>/apps/hive/warehouse</value> <description>location of default database for the warehouse</description> </property>Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.