当前访客身份：游客 [
当前位置：
在网上查了很多资料，都没有说的很清楚的。
我知道在namenode上通过命令 hadoop jar 的方式来提交mapreduce任务，但是在正真程序执行的时候不会是用手工去执行hadoop命令吧！
最近查了很多资料，竟然没有这方面的资料，我就想知道正真项目中是怎么允许一个mapreduce的。
共有4个答案
<span class="a_vote_num" id="a_vote_num_
对hadoop了解不太多，猜想吧，把要执行的文件传到namenode上，然后用ssh来执行hadoop jar。为了以后方便一点，应该可以自己写个脚本吧（虽然我不会写）
<span class="a_vote_num" id="a_vote_num_
hadoop2 支持 远程提交的，网上搜索有很多资料，不知道你是怎么搜的。随便找了几个链接：
&http://blog.csdn.net/mercedesqq/article/details/
/blog/1879442
/blog/aee4a.html
我们实际应用在windows环境下提交MR的jar包执行有问题，是hadoop(2.3)的bug， 需要打补丁，linux环境下没有任何问题。
<span class="a_vote_num" id="a_vote_num_
1.把服务器的hadoop配置拷贝下到本地，记得配置hosts指向namenode和resourcemanager(也许是jobtracker),本地执行hadoop jar 的方式来提交mapreduce任务
2.在命令行指定,-fs hdfs:://namenode -jt &jobtracker之类的命令行。
推荐用第一种方法。
<span class="a_vote_num" id="a_vote_num_
可以通过很多方式运行Job，比如shell脚本等
如果是定期任务可以做一个schedule，定期调用shell
比如aws上采用datapipeline来自定义运行Job的方式。
更多开发者职位上
有什么技术问题吗？
二号铺的其它问题
类似的话题1. job.properties
nameNode=hdfs://192.168.130.54:19000
jobTracker=192.168.130.54:19001
queueName=default
#HDFS上的目录
oozie.wf.application.path=/tmp/oozie/conf
oozie.libpath=/tmp/oozie/lib
inputDir=/tmp/oozie/input
outputDir=/tmp/oozie/output/
2. workflow.xml
&workflow-app xmlns="uri:oozie:workflow:0.1" name="map-reduce-wf"&
&start to="wordcount"/&
&action name="wordcount"&
&map-reduce&
&job-tracker&${jobTracker}&/job-tracker&
&name-node&${nameNode}&/name-node&
&delete path="${outputDir}"/&
&/prepare&
&configuration&
&property&
&name&mapred.mapper.new-api&/name&
&value&true&/value&
&/property&
&property&
&name&mapred.reducer.new-api&/name&
&value&true&/value&
&/property&
&property&
&name&mapred.job.queue.name&/name&
&value&${queueName}&/value&
&/property&
&property&
&name&mapreduce.mapper.class&/name&
&value&org.myorg.WordCount$WordCountMapper&/value&
&/property&
&property&
&name&mapreduce.reducer.class&/name&
&value&org.myorg.WordCount$WordCountReducer&/value&
&/property&
&property&
&name&mapred.output.key.class&/name&
&value&org.apache.hadoop.io.Text&/value&
&/property&
&property&
&property&
&name&mapred.output.value.class&/name&
&value&org.apache.hadoop.io.IntWritable&/value&
&/property&
&property&
&name&mapred.map.tasks&/name&
&value&1&/value&
&/property&
&property&
&name&mapred.input.dir&/name&
&value&${inputDir}&/value&
&/property&
&property&
&name&mapred.output.dir&/name&
&value&${outputDir}&/value&
&/property&
&/configuration&
&/map-reduce&
&ok to="end"/&
&error to="fail"/&
&kill name="fail"&
&message&Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]&/message&
&end name="end"/&
&/workflow-app&
&&&&&&&&&&&&&&&&&&&&
3. HDFS目录结构
/tmp
/tmp/oozie
/tmp/oozie/conf
/tmp/oozie/conf/workflow.xml
/tmp/oozie/conf/job.properties
/tmp/oozie/input
/tmp/oozie/input/data.txt
/tmp/oozie/lib
/tmp/oozie/lib/wordcount.jar
4. 运行：bin/ooize job -config ../map-reduce/job.properties -run
查看oozie.log，会抛出异常如下：
16:26:41,546
INFO ActionStartCommand:525 - USER[hmail] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[-oozie-hmai-W] ACTION[-oozie-hmai-W@wordcount] Next Retry, Attempt Number [3] in [60,000] milliseconds
16:27:41,583
WARN ActionStartCommand:528 - USER[hmail] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[-oozie-hmai-W] ACTION[-oozie-hmai-W@wordcount] [***-oozie-hmai-W@wordcount***]In call()....status=START_RETRY
16:27:41,776
WARN ActionStartCommand:528 - USER[hmail] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[-oozie-hmai-W] ACTION[-oozie-hmai-W@wordcount] Error starting action [wordcount]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Call to /192.168.130.54:19000 failed on local exception: java.io.EOFException]
org.apache.oozie.action.ActionExecutorException: JA009: Call to /192.168.130.54:19000 failed on local exception: java.io.EOFException
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:361)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:297)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:350)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:492)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:582)
at org.mand.wf.ActionStartCommand.call(ActionStartCommand.java:128)
at org.mand.wf.ActionStartCommand.execute(ActionStartCommand.java:249)
at org.mand.wf.ActionStartCommand.execute(ActionStartCommand.java:47)
at org.mand.call(Command.java:202)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:132)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Call to /192.168.130.54:19000 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1110)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy22.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111)
at org.apache.hadoop.hdfs.DFSClient.&init&(DFSClient.java:213)
at org.apache.hadoop.hdfs.DFSClient.&init&(DFSClient.java:180)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
at org.apache.oozie.service.KerberosHadoopAccessorService$4.run(KerberosHadoopAccessorService.java:227)
at org.apache.oozie.service.KerberosHadoopAccessorService$4.run(KerberosHadoopAccessorService.java:222)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.oozie.service.KerberosHadoopAccessorService.addFileToClassPath(KerberosHadoopAccessorService.java:222)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:276)
... 11 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
5. 查了一下这个异常，网上有说是因为因为服务器端的hadoop和客户端的不一致，我的oozie和hadoop版本分别是hadoop-0.20.2，oozio是oozie-2.3.2-cdh3u2，oozie中所包含的hadoop-core也是0.20.2版本的，应该不是这个问题吧，求大侠解救
Hi,xianneng.lin:
这个问题后来解决了吗，我也遇到同样的问题，能否共享一下解决方法？谢谢。
已解决问题
未解决问题我使用3台Centos虚拟机搭建了一个Hadoop2.6的集群。希望在windows7上面使用IDEA开发mapreduce程序，然后提交的远程的Hadoop集群上执行。经过不懈的google终于搞定
开始我使用hadoop的eclipse插件来执行job，竟然成功了，后来发现mapreduce是在本地执行的，根本没有提交到集群上。我把hadoop的4个配置文件加上后就开始出现了问题。
1：org.apache.hadoop.util.Shell$ExitCodeException:&/bin/bash:&line&0:&fg:&no&job&control&
网上说要修改源码，在Hadoop2.6已经合并了那个补丁。这个错误怎么解决的也忘记了
2：Stack trace: ExitCodeException exitCode=1:
3：Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
4：Error:&java.lang.RuntimeException:&java.lang.ClassNotFoundException:&Class&WordCount$Map&not&found
按照我的步骤走，这些问题都能解决，我使用的IDE是IDEA
1：复制Hadoop的4个配置文件放到src目录下面：core-site.xml,hdfs-site.xml,log4j.properties,mapred-site.xml,yarn-site.xml
2:配置mapred-site.xml
&configuration&
&property&
&name&mapreduce.framework.name&/name&
&value&yarn&/value&
&/property&
&property&
&name&mapred.remote.os&/name&
&value&Linux&/value&
&/property&
&property&
&name&mapreduce.app-submission.cross-platform&/name&
&value&true&/value&
&/property&
&property&
&name&mapreduce.application.classpath&/name&
/opt/hadoop-2.6.0/etc/hadoop,
/opt/hadoop-2.6.0/share/hadoop/common/*,
/opt/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*
&/property&
&property&
&name&mapreduce.jobhistory.address&/name&
&value&master:10020&/value&
&/property&
&property&
&name&mapreduce.jobhistory.webapp.address&/name&
&value&master:19888&/value&
&/property&
&/configuration&
注意mapreduce.application.classpath一定是绝对路径，不要搞什么$HADOOP_HOME,我这里反正是报错的
3：修改yarn-site.xml
&configuration&
&!-- Site specific YARN configuration properties --&
&property&
&name&yarn.nodemanager.aux-services&/name&
&value&mapreduce_shuffle&/value&
&/property&
&property&
&name&yarn.resourcemanager.address&/name&
&value&master:8032&/value&
&/property&
&property&
&name&yarn.application.classpath&/name&
/opt/hadoop-2.6.0/etc/hadoop,
/opt/hadoop-2.6.0/share/hadoop/common/*,
/opt/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*
&/property&
&/configuration&
注意yarn.application.classpath一定是绝对路径，不要搞什么$HADOOP_HOME
4:看下我的代码
package com.gaoxing.
import java.io.IOE
import java.security.PrivilegedExceptionA
import java.util.StringT
import org.apache.hadoop.conf.C
import org.apache.hadoop.fs.P
import org.apache.hadoop.io.IntW
import org.apache.hadoop.io.T
import org.apache.hadoop.mapreduce.J
import org.apache.hadoop.mapreduce.M
import org.apache.hadoop.mapreduce.R
import org.apache.hadoop.mapreduce.lib.input.FileInputF
import org.apache.hadoop.mapreduce.lib.output.FileOutputF
import org.apache.hadoop.security.UserGroupI
import org.apache.hadoop.util.GenericOptionsP
public class WordCount {
//继承mapper接口，设置map的输入类型为&Object,Text&
//输出类型为&Text,IntWritable&
public static class Map extends Mapper&Object,Text,Text,IntWritable&{
//one表示单词出现一次
private static IntWritable one = new IntWritable(1);
//word存储切下的单词
private Text word = new Text();
public void map(Object key,Text value,Context context) throws IOException,InterruptedException{
//对输入的行切词
StringTokenizer st = new StringTokenizer(value.toString());
while(st.hasMoreTokens()){
word.set(st.nextToken());//切下的单词存入word
context.write(word, one);
//继承reducer接口，设置reduce的输入类型&Text,IntWritable&
//输出类型为&Text,IntWritable&
public static class Reduce extends Reducer&Text,IntWritable,Text,IntWritable&{
//result记录单词的频数
private static IntWritable result = new IntWritable();
public void reduce(Text key,Iterable&IntWritable& values,Context context) throws IOException,InterruptedException{
int sum = 0;
//对获取的&key,value-list&计算value的和
for(IntWritable val:values){
sum += val.get();
//将频数设置到result
result.set(sum);
//收集结果
context.write(key, result);
* @param args
public static void main(String[] args) throws Exception{
Configuration conf = new Configuration();
// conf.set("mapred.remote.os","Linux");
// conf.set("yarn.resourcemanager.address","master:8032");
// conf.set("mapreduce.framework.name","yarn");
conf.set("mapred.jar","D:\\IdeaProjects\\hadooplearn\\out\\artifacts\\hadoo.jar");
//conf.set("mapreduce.app-submission.cross-platform","true");
Job job = Job.getInstance(conf);
job.setJobName("test");
//配置作业各个类
job.setJarByClass(WordCount.class);
job.setMapperClass(Map.class);
job.setCombinerClass(Reduce.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path("hdfs://master:9000/tmp/hbase-env.sh"));
FileOutputFormat.setOutputPath(job, new Path("hdfs://master:9000/tmp/out11"));
System.exit(job.waitForCompletion(true) ? 0 : 1);
conf.set("mapred.jar","D:\\IdeaProjects\\hadooplearn\\out\\artifacts\\hadoo.jar");这是最重要的一句，不然会报上面第4个问题
IDEA中有个功能就是编译的时候打包：
阅读(...) 评论()&&国之画&&&& &&&&&&
&& &&&&&&&&&&&&&&&&&&
鲁ICP备号-4
打开技术之扣，分享程序人生！