WASB access

[emlyn]

I've used the template in core-site.xml to access containers in Azure Storage before, but when I tried it again recently it didn't work. I found I had to replace the following section:

<property>
  <name>fs.AbstractFileSystem.wasb.Impl</name>
  <value>org.apache.hadoop.fs.azure.Wasb</value>
</property>

with this:

  <property>
    <name>fs.wasbs.impl</name>
    <value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
  </property>

I'm not sure if this is due to a change in Spark, Hadoop or something else. It might be worth adding this to the template and/or updating the docs.

[jafreck]

Which version of Spark are you using?

[emlyn]

I believe we're using Spark 2.3.1

[jpyrett]

Agree with emlyn's solution. You must restart the cluster after changing local core-site.xml worked after that. Thanks emlyn. Yes, the docs need updated