Tag: cloudera cdh
错误：java.lang.IllegalArgumentException：即使使用变通方法，比较方法也违反了其一般合同
我已经花了两天时间来缩短这个错误，即使我尝试了几个stackoverflowpost“-Djava.util.Arrays.useLegacyMergeSort = true”中建议的解决方法，但它也不起作用。 这是我的命令及其返回错误的详细信息： 命令： hadoop jar CloudBrush.jar -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -reads /Ec10k -asm Ec10k_Brush -k 21 -readlen 36 错误： Error: java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeHi(TimSort.java:895) at java.util.TimSort.mergeAt(TimSort.java:512) at java.util.TimSort.mergeCollapse(TimSort.java:437) at java.util.TimSort.sort(TimSort.java:241) at java.util.Arrays.sort(Arrays.java:1512) at java.util.ArrayList.sort(ArrayList.java:1454) at java.util.Collections.sort(Collections.java:175) at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252) at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at […]
运行hbase MR作业时，我的cdh5.2集群会出现FileNotFoundException
我的cdh5.2群集运行hbase MR作业时出现问题。 例如，我将hbase类路径添加到hadoop类路径中： vi /etc/hadoop/conf/hadoop-env.sh 添加行： export HADOOP_CLASSPATH=”/usr/lib/hbase/bin/hbase classpath:$HADOOP_CLASSPATH” 当我运行时： hadoop jar /usr/lib/hbase/hbase-server-0.98.6-cdh5.2.1.jar rowcounter “mytable” 我得到以下exception： 14/12/09 03:44:02 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://clusterName/usr/lib/hbase/lib/hbase-client-0.98.6-cdh5.2.1.jar Exception in thread “main” java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at […]
使用–jars的spark-submit yarn-cluster不起作用？
我试图通过以下命令向CDH纱线集群提交火花作业 我已经尝试了几种组合，但一切都行不通…我现在所有的poi jar都位于我的本地/ root，以及HDFS / user / root / lib，因此我尝试了以下 spark-submit –master yarn-cluster –class “ReadExcelSC” ./excel_sc.jar –jars /root/poi-3.12.jars, /root/poi-ooxml-3.12.jar, /root/poi-ooxml-schemas-3.12.jar spark-submit –master yarn-cluster –class “ReadExcelSC” ./excel_sc.jar –jars file:/root/poi-3.12.jars, file:/root/poi-ooxml-3.12.jar, file:/root/poi-ooxml-schemas-3.12.jar spark-submit –master yarn-cluster –class “ReadExcelSC” ./excel_sc.jar –jars hdfs://mynamenodeIP:8020/user/root/poi-3.12.jars,hdfs://mynamenodeIP:8020/user/root/poi-ooxml-3.12.jar,hdfs://mynamenodeIP:8020/user/root/poi-ooxml-schemas-3.12.jar 如何将jar传播到所有群集节点？ 因为以上都没有工作，并且工作仍然以某种方式无法引用该类，因为我不断得到相同的错误： java.lang.NoClassDefFoundError: org/apache/poi/ss/usermodel/WorkbookFactory 相同的命令与“–master local”一起使用 ，没有指定–jars，因为我已将我的jar复制到/ opt / cloudera / parcels / CDH / lib / […]
CDH5.2：MR，无法初始化任何输出收集器
Cloudera CDH5.2快速启动VM Cloudera Manager显示所有节点state = GREEN 我在Eclipse上做了一个MR工作，包括Build Path中的所有相关cloudera jar：avro-1.7.6-cdh5.2.0.jar，avro-mapred-1.7.6-cdh5.2.0-hadoop2.jar，hadoop-common -2.5.0-cdh5.2.0.jar，hadoop-mapreduce-client-core-2.5.0-cdh5.2.0.jar 我做了以下工作 hadoop jar jproject1.jar avro00.AvroUserPrefCount -libjars ${LIBJARS} avro/00/in avro/00/out 我得到以下错误，是Java堆问题，任何评论？ 先感谢您 14/11/14 01:02:40 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032 14/11/14 01:02:43 INFO input.FileInputFormat: Total input paths to process : 1 14/11/14 01:02:43 INFO mapreduce.JobSubmitter: number of splits:1 14/11/14 01:02:44 INFO mapreduce.JobSubmitter: Submitting tokens for […]
