如何在动作书中的mahout中运行示例

我试图在第7章中运行hello world示例。我在eclipse中创建了以下内容，然后将其打包到jar中： –

package com.mycode.mahout import java.io.File; import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.SequenceFile; import org.apache.hadoop.io.Text; import org.apache.mahout.clustering.WeightedVectorWritable; import org.apache.mahout.clustering.kmeans.Cluster; import org.apache.mahout.clustering.kmeans.KMeansDriver; import org.apache.mahout.common.distance.EuclideanDistanceMeasure; import org.apache.mahout.math.RandomAccessSparseVector; import org.apache.mahout.math.Vector; import org.apache.mahout.math.VectorWritable; public class SimpleKMeansClustering { public static final double[][] points = { {1, 1}, {2, 1}, {1, 2}, {2, 2}, {3, 3}, {8, 8}, {9, 8}, {8, 9}, {9, 9}}; public static void writePointsToFile(List points, String fileName, FileSystem fs, Configuration conf) throws IOException { Path path = new Path(fileName); SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path, LongWritable.class, VectorWritable.class); long recNum = 0; VectorWritable vec = new VectorWritable(); for (Vector point : points) { vec.set(point); writer.append(new LongWritable(recNum++), vec); } writer.close(); } public static List getPoints(double[][] raw) { List points = new ArrayList(); for (int i = 0; i < raw.length; i++) { double[] fr = raw[i]; Vector vec = new RandomAccessSparseVector(fr.length); vec.assign(fr); points.add(vec); } return points; } public static void main(String args[]) throws Exception { int k = 2; List vectors = getPoints(points); File testData = new File("testdata"); if (!testData.exists()) { testData.mkdir(); } testData = new File("testdata/points"); if (!testData.exists()) { testData.mkdir(); } Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); writePointsToFile(vectors, "testdata/points/file1", fs, conf); Path path = new Path("testdata/clusters/part-00000"); SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path, Text.class, Cluster.class); for (int i = 0; i < k; i++) { Vector vec = vectors.get(i); Cluster cluster = new Cluster(vec, i, new EuclideanDistanceMeasure()); writer.append(new Text(cluster.getIdentifier()), cluster); } writer.close(); KMeansDriver.run(conf, new Path("testdata/points"), new Path("testdata/clusters"), new Path("output"), new EuclideanDistanceMeasure(), 0.001, 10, true, false); SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path("output/" + Cluster.CLUSTERED_POINTS_DIR + "/part-m-00000"), conf); IntWritable key = new IntWritable(); WeightedVectorWritable value = new WeightedVectorWritable(); while (reader.next(key, value)) { System.out.println(value.toString() + " belongs to cluster " + key.toString()); } reader.close(); } } 

我把它打包成myjob.jar

现在我该如何在我的集群上执行此操作？

我试过以下： –

 hadoop jar myjob.jar com.mycode.mahout.SimpleKMeansClustering java -jar myjob.jar java -cp myjob.jar 

我得到了以下错误： –

  [root@node1 tmp]# hadoop jar mahoutfirst.jar com.mahout.emc.SimpleKMeansClustering Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/Vector` at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.hadoop.util.RunJar.main(RunJar.java:201) Caused by: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 3 more 

请建议运行使用mahout编写的代码的正确方法是什么。

Answer 1

虽然这很晚但我遇到了类似的问题，并且以下方法对我有用，因为我不想使用maven：

1）转到mahout安装目录并查找* job.jar as

 ls /usr/lib/mahout/ conf lib mahout-core-0.5-cdh3u3-job.jar mahout-examples-0.5-cdh3u3-job.jar mahout-taste-webapp-0.5-cdh3u3.war 

2）将mahout-examples-0.5-cdh3u3-job.jar复制到代码所在的目录

3）使用Mahout提供的“job”JAR文件。 它打包了所有依赖项。 您还需要将类添加到其中。 当您使用hadoop和mahout库编译您的类时，您已准备好.class文件。

4）将您的类文件添加到您的目录中的作业jar mahout-core-0.5-cdh3u3-job.jar：

 jar uf mahout-core-0.5-cdh3u3-job.jar SimpleKMeansClustering.class 

4）使用您的代码运行hadoop jar：

 hadoop jar mahout-core-0.5-cdh3u3-job.jar SimpleKMeansClustering 

5）在map-reduce作业结束时，您可以看到：

 1.0: [1.000, 1.000] belongs to cluster 0 1.0: [2.000, 1.000] belongs to cluster 0 1.0: [1.000, 2.000] belongs to cluster 0 1.0: [2.000, 2.000] belongs to cluster 0 1.0: [3.000, 3.000] belongs to cluster 0 1.0: [8.000, 8.000] belongs to cluster 1 1.0: [9.000, 8.000] belongs to cluster 1 1.0: [8.000, 9.000] belongs to cluster 1 1.0: [9.000, 9.000] belongs to cluster 1 

Answer 2

看看上面发现的非类定义exception，似乎你可能需要在你的Hadoop作业中包含Mahout相关的jar（mahout-core.jar，我猜）。

要将jar传递给整个集群中的映射器，您可能需要使用DistributedCache或-libjar Hadoop选项。 这里解释了后者的想法。