hadoop2.7分布式安装_百度文库
两大类热门资源免费畅读
续费一年阅读会员，立省24元！
hadoop2.7分布式安装
上传于||文档简介
&&h​a​d​o​o​p..最​新​安​装​文​档​,​分​布​式​安​装
阅读已结束，如果下载本文需要使用0下载券
想免费下载更多文档？
定制HR最喜欢的简历
下载文档到电脑，查找使用更方便
还剩2页未读，继续阅读
定制HR最喜欢的简历
你可能喜欢webseven 的BLOG
用户名：webseven
文章数：60
访问量：3990
注册日期：
阅读量：5863
阅读量：12276
阅读量：333082
阅读量：1038728
51CTO推荐博文
1.环境准备: 安装Centos6.5的操作系统 下载hadoop2.7版本的软件 wget http://124.205.69.132/files/626A/mirrors./apache/hadoop/common/stable/hadoop-2.7.1.tar.gz 下载jdk1.87版本的软件 wget /otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz?AuthParam=_ab1a6a92468aba5cd4d092d02.修改/etc/hosts文件及配置互信: 在/etc/hosts文件中增加如下内容:
192.168.1.61 host61
192.168.1.62 host62
192.168.1.63 host63 配置好各服务器之间的ssh互信3.添加用户,解压文件并配置环境变量:
useradd hadoop passwd hadoop
tar -zxvf hadoop-2.7.1.tar.gz mv hadoop-2.7.1 /usr/local ln -s hadoop-2.7.1 hadoop chown -R hadoop:hadoop hadoop-2.7.1
tar -zxvf jdk-8u60-linux-x64.tar.gz mv jdk1.8.0_60 /usr/local ln -s jdk1.8.0_60 jdk chown -R root:root jdk1.8.0_60 echo 'export JAVA_HOME=/usr/local/jdk' &&/etc/profile
echo 'export PATH=/usr/local/jdk/bin:$PATH' &/etc/profile.d/java.sh 4.修改hadoop配置文件: 1)修改hadoop-env.sh文件:
cd /usr/local/hadoop/etc/hadoop/hadoop-env.sh
sed -i 's%#export JAVA_HOME=${JAVA_HOME}%export JAVA_HOME=/usr/local/jdk%g' hadoop-env.sh
2)修改core-site.xml,在最后添加如下内容:
&configuration& &
&property&
&name&fs.default.name&/name&
&value&hdfs://host61:9000/&/value&
&/property&
&property&
&name&hadoop.tmp.dir&/name&
&value&/home/hadoop/temp&/value&
&/property&
&/configuration& & 3)修改hdfs-site.xml文件:
&configuration& & &
&property& & &
&name&dfs.replication&/name& & &
&value&3&/value& & &
&/property& & &
&/configuration& 4)修改mapred-site.xml
&configuration& & &
&property& & &
&name&mapred.job.tracker&/name& & &
&value&host61:9001&/value& & &
&/property& & &
&/configuration& 5)配置masters
host61 6)配置slaves
host635.用同样的方式配置host62及host636.格式化分布式文件系统 /usr/local/hadoop/bin/hadoop namenode format 7.替换hadoop的库文件: mv /usr/local/hadoop/lib/native /usr/local/hadoop/lib/native_old 将编译好的hadoop文件下的lib/native文件夹复制过来; 8.运行hadoop 1)/usr/local/hadoop/sbin/start-dfs.sh 2)/usr/local/hadoop/sbin/start-yarn.sh9.检查: [root@host61 sbin]# jps 4532 ResourceManager 4197 NameNode 4793 Jps 4364 SecondaryNameNode
[root@host62 ~]# jps 32052 DataNode 32133 NodeManager 32265 Jps
[root@host63 local]# jps 6802 NodeManager 6963 Jps 6717 DataNode 10.通过web了解hadoop: namenode的信息: http://192.168.1.61:50070/
secondnamenode的信息: http://192.168.1.61:50090/
datanode的信息: http://192.168.1.62:50075/
11.测试 echo "this is the first file" &/tmp/mytest1.txt echo "this is the second file" &/tmp/mytest2.txt
cd /usr/local/hadoop/ [hadoop@host61 bin]$ ./hadoop fs -mkdir /in
[hadoop@host61 bin]$ ./hadoop fs -put /tmp/mytest*.txt /in
[hadoop@host61 bin]$ ./hadoop fs -ls /in Found 2 items -rw-r--r-- & 3 hadoop supergroup & & & & 23
18:45 /in/mytest1.txt -rw-r--r-- & 3 hadoop supergroup & & & & 24
18:45 /in/mytest2.txt [hadoop@host61 hadoop]$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar &wordcount /in /out 15/10/02 18:53:30 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 15/10/02 18:53:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 15/10/02 18:53:34 INFO input.FileInputFormat: Total input paths to process : 2 15/10/02 18:53:35 INFO mapreduce.JobSubmitter: number of splits:2 15/10/02 18:53:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local_0001 15/10/02 18:53:40 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 15/10/02 18:53:40 INFO mapreduce.Job: Running job: job_local_0001 15/10/02 18:53:40 INFO mapred.LocalJobRunner: OutputCommitter set in config null 15/10/02 18:53:40 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/10/02 18:53:40 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 15/10/02 18:53:41 INFO mapred.LocalJobRunner: Waiting for map tasks 15/10/02 18:53:41 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_ 15/10/02 18:53:41 INFO mapreduce.Job: Job job_local_0001 running in uber mode : false 15/10/02 18:53:41 INFO mapreduce.Job: &map 0% reduce 0% 15/10/02 18:53:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/10/02 18:53:41 INFO mapred.Task: &Using ResourceCalculatorProcessTree : [ ] 15/10/02 18:53:41 INFO mapred.MapTask: Processing split: hdfs://host61:9000/in/mytest2.txt:0+24 15/10/02 18:53:51 INFO mapred.MapTask: (EQUATOR) 0 kvi 857584) 15/10/02 18:53:51 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/10/02 18:53:51 INFO mapred.MapTask: soft limit at
15/10/02 18:53:51 INFO mapred.MapTask: bufstart = 0; bufvoid =
15/10/02 18:53:51 INFO mapred.MapTask: kvstart = ; length = 6553600 15/10/02 18:53:51 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 15/10/02 18:53:52 INFO mapred.LocalJobRunner:& 15/10/02 18:53:52 INFO mapred.MapTask: Starting flush of map output 15/10/02 18:53:52 INFO mapred.MapTask: Spilling map output 15/10/02 18:53:52 INFO mapred.MapTask: bufstart = 0; bufend = 44; bufvoid =
15/10/02 18:53:52 INFO mapred.MapTask: kvstart = 857584); kvend = 857520); length = 17/6553600 15/10/02 18:53:52 INFO mapred.MapTask: Finished spill 0 15/10/02 18:53:52 INFO mapred.Task: Task:attempt_local_0001_m_ is done. And is in the process of committing 15/10/02 18:53:53 INFO mapred.LocalJobRunner: map 15/10/02 18:53:53 INFO mapred.Task: Task 'attempt_local_0001_m_' done. 15/10/02 18:53:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_ 15/10/02 18:53:53 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_ 15/10/02 18:53:53 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/10/02 18:53:53 INFO mapred.Task: &Using ResourceCalculatorProcessTree : [ ] 15/10/02 18:53:53 INFO mapred.MapTask: Processing split: hdfs://host61:9000/in/mytest1.txt:0+23 15/10/02 18:53:53 INFO mapreduce.Job: &map 100% reduce 0% 15/10/02 18:53:53 INFO mapred.MapTask: (EQUATOR) 0 kvi 857584) 15/10/02 18:53:53 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/10/02 18:53:53 INFO mapred.MapTask: soft limit at
15/10/02 18:53:53 INFO mapred.MapTask: bufstart = 0; bufvoid =
15/10/02 18:53:53 INFO mapred.MapTask: kvstart = ; length = 6553600 15/10/02 18:53:53 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 15/10/02 18:53:54 INFO mapred.LocalJobRunner:& 15/10/02 18:53:54 INFO mapred.MapTask: Starting flush of map output 15/10/02 18:53:54 INFO mapred.MapTask: Spilling map output 15/10/02 18:53:54 INFO mapred.MapTask: bufstart = 0; bufend = 43; bufvoid =
15/10/02 18:53:54 INFO mapred.MapTask: kvstart = 857584); kvend = 857520); length = 17/6553600 15/10/02 18:53:54 INFO mapred.MapTask: Finished spill 0 15/10/02 18:53:54 INFO mapred.Task: Task:attempt_local_0001_m_ is done. And is in the process of committing 15/10/02 18:53:54 INFO mapreduce.Job: &map 50% reduce 0% 15/10/02 18:53:54 INFO mapred.LocalJobRunner: map 15/10/02 18:53:54 INFO mapred.Task: Task 'attempt_local_0001_m_' done. 15/10/02 18:53:54 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_ 15/10/02 18:53:54 INFO mapred.LocalJobRunner: map task executor complete. 15/10/02 18:53:54 INFO mapred.LocalJobRunner: Waiting for reduce tasks 15/10/02 18:53:54 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_r_ 15/10/02 18:53:54 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/10/02 18:53:54 INFO mapred.Task: &Using ResourceCalculatorProcessTree : [ ] 15/10/02 18:53:54 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@ 15/10/02 18:53:55 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=, maxSingleShuffleLimit=, mergeThreshold=, ioSortFactor=10, memToMemMergeOutputsThreshold=10 15/10/02 18:53:55 INFO reduce.EventFetcher: attempt_local_0001_r_ Thread started: EventFetcher for fetching Map Completion Events 15/10/02 18:53:55 INFO mapreduce.Job: &map 100% reduce 0% 15/10/02 18:53:56 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local_0001_m_ decomp: 55 len: 59 to MEMORY 15/10/02 18:53:56 INFO reduce.InMemoryMapOutput: Read 55 bytes from map-output for attempt_local_0001_m_ 15/10/02 18:53:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -& map-output of size: 55, inMemoryMapOutputs.size() -& 1, commitMemory -& 0, usedMemory -&55 15/10/02 18:53:56 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local_0001_m_ decomp: 56 len: 60 to MEMORY 15/10/02 18:53:56 INFO reduce.InMemoryMapOutput: Read 56 bytes from map-output for attempt_local_0001_m_ 15/10/02 18:53:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -& map-output of size: 56, inMemoryMapOutputs.size() -& 2, commitMemory -& 55, usedMemory -&111 15/10/02 18:53:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning 15/10/02 18:53:56 INFO mapred.LocalJobRunner: 2 / 2 copied. 15/10/02 18:53:56 INFO reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs 15/10/02 18:53:57 INFO mapred.Merger: Merging 2 sorted segments 15/10/02 18:53:57 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 97 bytes 15/10/02 18:53:57 INFO reduce.MergeManagerImpl: Merged 2 segments, 111 bytes to disk to satisfy reduce memory limit 15/10/02 18:53:57 INFO reduce.MergeManagerImpl: Merging 1 files, 113 bytes from disk 15/10/02 18:53:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 15/10/02 18:53:57 INFO mapred.Merger: Merging 1 sorted segments 15/10/02 18:53:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 102 bytes 15/10/02 18:53:57 INFO mapred.LocalJobRunner: 2 / 2 copied. 15/10/02 18:53:57 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 15/10/02 18:53:59 INFO mapred.Task: Task:attempt_local_0001_r_ is done. And is in the process of committing 15/10/02 18:53:59 INFO mapred.LocalJobRunner: 2 / 2 copied. 15/10/02 18:53:59 INFO mapred.Task: Task attempt_local_0001_r_ is allowed to commit now 15/10/02 18:53:59 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_' to hdfs://host61:9000/out/_temporary/0/task_local_0001_r_000000 15/10/02 18:53:59 INFO mapred.LocalJobRunner: reduce & reduce 15/10/02 18:53:59 INFO mapred.Task: Task 'attempt_local_0001_r_' done. 15/10/02 18:53:59 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_r_ 15/10/02 18:53:59 INFO mapred.LocalJobRunner: reduce task executor complete. 15/10/02 18:53:59 INFO mapreduce.Job: &map 100% reduce 100% 15/10/02 18:53:59 INFO mapreduce.Job: Job job_local_0001 completed successfully 15/10/02 18:54:00 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=821850
FILE: Number of bytes written=1655956
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=118
HDFS: Number of bytes written=42
HDFS: Number of read operations=22
HDFS: Number of large read operations=0
HDFS: Number of write operations=5
Map-Reduce Framework
Map input records=2
Map output records=10
Map output bytes=87
Map output materialized bytes=119
Input split bytes=196
Combine input records=10
Combine output records=10
Reduce input groups=6
Reduce shuffle bytes=119
Reduce input records=10
Reduce output records=6
Spilled Records=20
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=352
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters&
Bytes Read=47
File Output Format Counters&
Bytes Written=42 [hadoop@host61 hadoop]$&
[hadoop@host61 hadoop]$ ./bin/hadoop fs -ls /out Found 2 items -rw-r--r-- & 3 hadoop supergroup & & & & &0
18:53 /out/_SUCCESS -rw-r--r-- & 3 hadoop supergroup & & & & 42
18:53 /out/part-r-00000 [hadoop@host61 hadoop]$ ./bin/hadoop fs -cat /out/_SUCCESS [hadoop@host61 hadoop]$ ./bin/hadoop fs -cat /out/part-r-00000 file 2 first 1 is 2 second 1 the 2 this 2 [hadoop@host61 hadoop]$& 12.至此hadoop的配置部署工作顺利完成; 本文出自 “” 博客，请务必保留此出处
了这篇文章
类别：┆阅读(0)┆评论(0)操作系统：在windows7下使用ubuntu-14.04.3-desktop-amd64
hadoop版本：hadoop-2.7.1
jdk版本：jdk-7u79-linux-x64.tar.gz
1.&&&&&修改主机名为master
$ sudo vim /etc/hostname
编辑hostname文件，在文件中输入master并保存该文件即可。重启系统。
注：值的指出的是，在其它Linux发行版中，并非都存在/etc/hostname文件。如Fedora发行版将主机名存放在/etc/sysconfig/network文件中。所以，修改主机名时应注意区分是哪种Linux发行版。
2.&&&&&修改hosts
$ sudo vim /etc/hosts
修改内容: 127.0.0.1 master
3.&&&&&配置SSH
此处不再讲述，请参考
4.&&&&& 安装配置hadoop-2.7.1
此处不再讲述，请参考
5.&&&&&在hadoop目录创建文件
$ mkdir dfs
$ mkdir dfs/name
$ mkdir dfs/data
6.&&&&&修改hadoop配置文件
1)&&&&&修改配置文件hadoop-env.sh，
$ sudo vim etc/hadoop/hadoop-env.sh
修改JAVA_HOME为我们安装的JAVA_HOME,
export JAVA_HOME=/usr/lib/java/jdk1.7.0_79
2)&&&&& 修改配置文件yarn-env.sh,
$ sudo wimetc/hadoop/yarn-env.sh
修改JAVA_HOME为我们安装的JAVA_HOME,
export JAVA_HOME=/usr/lib/java/jdk1.7.0_79
3)&&&&& 修改配置文件mapred-env.sh
$ sudo wimetc/hadoop/mapred-env.sh
修改JAVA_HOME为我们安装的JAVA_HOME,
exportJAVA_HOME=/usr/lib/java/jdk1.7.0_79
4)&&&&& 修改slaves文件
$ sudo vimetc/hadoop/slaves
修改 localhost为master
5)&&&&& 修改配置文件core-site.xml
$ sudo vimetc/hadoop/core-site.xml
修改内容为:
&configuration&
&& &property&
&&&&&&& &name&fs.defaultFS&/name&
&&&&&&&&value&hdfs://master:9000&/value&
&&& &/property&
&&& &property&
&&&&&&& &name&hadoop.tmp.dir&/name&
&&&&&&& &value&/home/tongwei/Work/Dev/Hadoop/hadoop-2.7.1/tmp&/value&
&&&&&&& &description&A base of othertemporary directories&/description&
&& &/property&
&/configuration&
core-site.xml各项配置可参考：
6)&&&&& 修改配置文件 hdfs-site.xml
$ sudo vimetc/hadoop/hdfs-site.xml
修改内容为:
&configuration&
&& &property&
&&&&&&&&name&dfs.replication&/name&
&&&&&&& &value&1&/value&
&& &/property&
&& &property&
&&&&&&&&name&dfs.namenode.name.dir&/name&
&&&&&&& &value&/home/tongwei/Work/Dev/Hadoop/hadoop-2.7.1/dfs/name&/value&
&&& &/property&
&&& &property&
&&&&&&&&name&dfs.datanode.data.dir&/name&
&&&&&&&&value&/home/tongwei/Work/Dev/Hadoop/hadoop-2.7.1/dfs/data&/value&
&& &/property&
&/configuration&
hdfs-site.xml各项配置可参考：
7)&&&&& 修改配置文件 mapred-site.xml
$ sudo vimetc/hadoop/mapred-site.xml
修改内容为:
&configuration&
&&& &property&
&&&&&&&&name&mapreduce.framework.name&/name&
&&&&&&& &value&yarn&/value&
&&& &/property&
&/configuration&
mapred-site.xml各项配置可参考：
8)&&&&& 配置文件yarn-site.xml
$ sudo vimetc/hadoop/yarn-site.xml
修改内容为:
&configuration&
&&& &property&
&&&&&&&&name&yarn.resourcemanager.hostname&/name&
&&&&&&& &value&master&/value&
&& &/property&
&&& &property&
&&&&&&& &name&yarn.nodemanager.aux-services&/name&
&&&&&&&&value&mapreduce_shuffle&/value&
&&& &/property&
&/configuration&
yarn-site文件配置的各项内容可参考：
也可以增加spark_shuffle,配置如下
&property&
&&name&yarn.nodemanager.aux-services&/name&
&&value&mapreduce_shuffle,spark_shuffle&/value&
&/property&
&property&
& &name&yarn.nodemanager.aux-services.mapreduce_shuffle.class&/name&
&&value&org.apache.hadoop.mapred.ShuffleHandler&/value&
&/property&
&property&
&&name&yarn.nodemanager.aux-services.spark_shuffle.class&/name&
&&value&org.apache.spark.network.yarn.YarnShuffleService&/value&
&/property&
个人认为 当提交hadoop MR 就启用,mapreduce_shuffle,当提交spark作业 就使用spark_shuffle,但个人感觉spark_shuffle 效率一般,shuffle是很大瓶颈,还有 如果你使用spark_shuffle 你需要把spark-yarn_2.10-1.4.1.jar 这个jar copy 到HADOOP_HOME/share/hadoop/lib下 ,否则 hadoop 运行报错 class not find exeception.
7.&&&&&启动并验证hadoop伪分布式
1)&&&&&格式化hdfs文件系统
$ bin/hadoop namenode –format
2)&&&&&启动hdfs
$ sbin/start-dfs.sh
此刻我们发现在master上启动了NameNode、DataNode、SecondaryNameNode
此刻通过web控制台查看hdfs，
3)&&&&&启动yarn
$ sbin/start-yarn.sh
使用jps命令可以发现master机器启动了ResourceManager进程
5049 NodeManager
4388 NameNode
4509 DataNode
4926 ResourceManager
4712 SecondaryNameNode
PS:我们上传一个文件到hdfs吧：
$ bin/hadoop fs -mkdir -p dfs/data/test
$ bin/hadoop fs -put README.txt dfs/data/test
$ bin/hadoop fs -text dfs/data/test/README.txt
$ bin/hadoop fs命令参考
hadoop web控制台页面的端口整理：
50070：hdfs文件管理
8088：ResourceManager
8042：NodeManager
19888：JobHistory(使用“mr-jobhistory-daemon.sh”来启动JobHistoryServer)
参考知识库
* 以上用户言论只代表其个人观点，不代表CSDN网站的观点或立场
访问：3470次
排名：千里之外
(3)(1)(1)(3)哪位大神知道从安装hadoop2.7.1到配置文件的所有正确流程_百度知道