Spring Boot YARN无法在Hadoop上运行2.8.0客户端无法访问DataNode
我正在尝试运行Spring Boot YARN示例（Windows上的https://spring.io/guides/gs/yarn-basic/ ）。 在application.yml我将fsUri和resourceManagerHost更改为指向我的VM的主机192.168... 但是，当我试图运行应用程序Exceprion出现时：
DFSClient: Exception in createBlockOutputStream java.net.ConnectException: Connection timed out: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1508) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1284) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449) [2017-05-27 19:59:49.570] boot - 7728 INFO [Thread-5] --- DFSClient: Abandoning BP-646365587-10.0.2.15-1495898351938:blk_1073741830_1006 [2017-05-27 19:59:49.602] boot - 7728 INFO [Thread-5] --- DFSClient: Excluding datanode DatanodeInfoWithStorage[10.0.2.15:50010,DS-f909ec7a-8374-4cdd-9cfc-0e778810d98c,DISK] [2017-05-27 19:59:49.647] boot - 7728 WARN [Thread-5] --- DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /app/gs-yarn-basic/gs-yarn-basic-container-0.1.0.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
这意味着无法从我的主机访问DataNode。 出于这个原因，我添加到hdfs-site.xml
dfs.client.use.datanode.hostname true Whether clients should use datanode hostnames when connecting to datanodes.
但它仍然抛出exception。
我的VM上运行了Hadoop 2.8.0。 这是conf。 文件：
核心的site.xml
fs.defaultFS hdfs://0.0.0.0:9000
HDFS-site.xml中
dfs.replication 1 dfs.namenode.name.dir /usr/local/hadoop/hadoop-2.8.0/data/namenode dfs.datanode.data.dir /usr/local/hadoop/hadoop-2.8.0/data/datanode dfs.permissions.enabled false dfs.client.use.datanode.hostname true Whether clients should use datanode hostnames when connecting to datanodes.
mapred-site.xml中
mapreduce.framework.name yarn
纱的site.xml
yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.scheduler.maximum-allocation-mb 8192 yarn.nodemanager.resource.memory-mb 8192 yarn.nodemanager.disk-health-checker.max-disk-utilization-per- disk-percentage 99
您的core-site.xml应指向Namenode地址，但当前它指向0.0.0.0 ，这意味着本地计算机上的所有地址。 这会产生不明确的结果，因为每台机器都应被视为Namenode 。
Namenode应该只是hadoop集群中的一个。
用Namenode的ip或hostname替换0.0.0.0应该可以解决您遇到的问题。
Spring将0.0.0.0:9000更改为[VM的IP]：在core-site.xml中为9000后连接到YARN。 感谢@RameshMaharjan