Search code examples
hadoopknox-gateway

Issue configuring Apache KNOX gateway with WEBHDFS

I have installed Apache KNOX using Apache Ambari. Followed the below link for configuring the WEBHDFS with KNOX.KNOX is installed in address1. https://knox.apache.org/books/knox-0-13-0/user-guide.html#WebHDFS

While invoking webhdfs using the curl command curl -i -k -u guest:guest-password -X GET 'https://address1:8443/gateway/Andromd/webhdfs/v1/?op=LISTSTATUS'

it throws the following error

<title> 404 Not Found</title>
<p> Problem accessing /gateway//Andromd/webhdfs/v1/build-version.Reason:
<pre> Not Found </pre></p>

Andromd is the cluster name and added it under {GATEWAY_HOME}/conf/topologies/Andromd.xml. Configurations are as follows.

<service>
    <role>NAMENODE</role>
    <url>hdfs://address2:8020</url>
</service>
<service>
    <role>WEBHDFS</role>
    <url>http://address2:50070/webhdfs</url>
</service

Content of /etc/hadoop/conf/hdfs-site.xml is as follows.

<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.namenode.rpc-address</name>
    <value>address2:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>address2:50070</value>
</property>
<property>
    <name>dfs.https.namenode.https-address</name>
    <value>address2:50470</value>
</property>

Do let me know , is there anything missing from my side regarding the configuration.

Gateway.log content:

2017-10-13 18:24:24,586 ERROR digester3.Digester (Digester.java:parse(1652)) - An error occurred while parsing XML from '(already loaded from stream)', see nested exceptions
org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
        at org.apache.commons.digester3.Digester.parse(Digester.java:1642)
        at org.apache.commons.digester3.Digester.parse(Digester.java:1701)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopologyAttempt(DefaultTopologyService.java:124)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopology(DefaultTopologyService.java:100)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopologies(DefaultTopologyService.java:233)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.reloadTopologies(DefaultTopologyService.java:318)
        at org.apache.hadoop.gateway.GatewayServer.start(GatewayServer.java:312)
        at org.apache.hadoop.gateway.GatewayServer.startGateway(GatewayServer.java:231)
        at org.apache.hadoop.gateway.GatewayServer.main(GatewayServer.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
        at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
        at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101)
        at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
        at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46)
2017-10-13 18:24:24,587 ERROR hadoop.gateway (DefaultTopologyService.java:loadTopologies(250)) - Failed to load topology /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies/Andromeda.xml: org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.
2017-10-13 18:24:24,588 INFO  hadoop.gateway (GatewayServer.java:handleCreateDeployment(450)) - Loading topology admin from /usr/hdp/2.4.2.0-258/knox/bin/../data/deployments/admin.war.15f15c171c0
2017-10-13 18:24:24,793 INFO  hadoop.gateway (GatewayServer.java:start(315)) - Monitoring topologies in directory: /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies
2017-10-13 18:24:24,795 INFO  hadoop.gateway (GatewayServer.java:startGateway(232)) - Started gateway on port 8,443.

Added to above query, is necessary to include below tags in Andromd.xml,by replacing the sandbox.hortonworks.com with our cluster specific information.

<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.namenode.rpc-address</name>
    <value>sandbox.hortonworks.com:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>sandbox.hortonworks.com:50070</value>
</property>
<property>
    <name>dfs.https.namenode.https-address</name>
    <value>sandbox.hortonworks.com:50470</value>
</property>
Solution 
  • 2017-10-13 18:24:24,587 ERROR hadoop.gateway (DefaultTopologyService.java:loadTopologies(250)) - Failed to load topology /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies/Andromeda.xml: org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.

    Looking at this line it looks like your topology file Andromeda.xml is not well formatted (missing closing tags or some such) at lineNumber: 88.

    This is the reason why you are getting 404 because the topology is not being deployed. Check the logs after fixing the topology file and make sure there are no startup errors.