hadoop-006-完全分布式问题总结-白红宇

hadoop-006-完全分布式问题总结

阅读量：6684 次

发布时间：2019-06-25

本文共 9213 字，大约阅读时间需要 30 分钟。

一、运行mapreduce 任务，但是在yarn里看不到任务。

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /test /out1

我配置的resourceManager 为 http://192.168.31.136:8088

原因是在 mapred-site.xml 没有配置

<name>mapreduce.framework.name</name>

</property>

二、配置好问题一后，运行mr任务提示The auxService:mapreduce_shuffle does not exist

原因是在yarn-site.xml 没有配置yarn.nodemanager.aux-services 节点

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

报错如下

16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED

Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------------------------------

配置文件记录

已提前搭建好三个节点的zookeeper 小集群，配置可以实现 HDFS　ＨＡ和　ＹＡＲＮ　ＨＡ

一、hadoop-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

二、yarn-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

三、core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<name>fs.defaultFS</name>

<value>hdfs://mycluster</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/jxlgzwh/hadoop-2.7.2/data/jn</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/jxlgzwh/hadoop-2.7.2/data/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>master:2181,slave01:2181,slave02:2181</value>

</property>

</configuration>

四、hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<name>dfs.nameservices</name>

<value>mycluster</value>

</property>

<name>dfs.ha.namenodes.mycluster</name>

</property>

<name>dfs.namenode.rpc-address.mycluster.nn1</name>

<value>master:8020</value>

</property>

<name>dfs.namenode.rpc-address.mycluster.nn2</name>

<value>slave01:8020</value>

</property>

<name>dfs.namenode.http-address.mycluster.nn1</name>

<value>master:50070</value>

</property>

<name>dfs.namenode.http-address.mycluster.nn2</name>

<value>slave01:50070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://master:8485;slave01:8485;slave02:8485/mycluster</value>

</property>

<name>dfs.client.failover.proxy.provider.mycluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/jxlgzwh/.ssh/id_dsa</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.permissions.enabled</name>

<value>false</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

</configuration>

五、mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<name>mapreduce.framework.name</name>

</property>

</configuration>

六、slaves

192.168.31.136

192.168.31.130

192.168.31.229

七、yarn-site.xml

<?xml version="1.0"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<name>yarn.resourcemanager.ha.enabled</name>

</property>

<name>yarn.resourcemanager.cluster-id</name>

<value>cluster1</value>

</property>

<name>yarn.resourcemanager.ha.rm-ids</name>

</property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>master</value>

</property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>slave01</value>

</property>

<name>yarn.resourcemanager.webapp.address.rm1</name>

<value>master:8088</value>

</property>

<name>yarn.resourcemanager.webapp.address.rm2</name>

<value>slave01:8088</value>

</property>

<name>yarn.resourcemanager.zk-address</name>

<value>master:2181,slave01:2181,slave02:2181</value>

</property>

<name>yarn.resourcemanager.recovery.enabled</name>

</property>

<name>yarn.resourcemanager.zk-state-store.parent-path</name>

<value>/rmstore</value>

</property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

<name>yarn.nodemanager.recovery.enabled</name>

</property>

<name>yarn.nodemanager.address</name>

</property>

<name>yarn.nodemanager.recovery.dir</name>

<value>/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

八、配置 /etc/hosts 文件并配置 ssh免密码登录

192.168.31.136 master.com master

192.168.31.130 slave01

192.168.31.229 slave02

转载于:https://my.oschina.net/zhongwenhao/blog/796476

你可能感兴趣的文章

python django django-debug-toolbar 加载缓慢，不能使用。

Butterknife 的简单使用和配合 Butterknife的插件 Zelezny

查看>>

Magento利用input type=”file”上传图片

查看>>

Android音频开发（4）：如何存储和解析wav文件

查看>>

Handler延迟事件使用

查看>>

【DG】Oracle 19c使用dbca来搭建物理DG

查看>>

StringBuilder 、StringBuffer 、 String

SpringMVC Hello World 实例

【转】js之iframe子页面与父页面通信

需求管理工具比较 Doors_Requistie Pro_RDM

查看>>

centos+php+nginx的php.ini无法加载的问题

查看>>