将文件从HDFS复制到本地计算机

我在尝试从HDFS文件系统“下载”文件到我的本地系统时遇到问题。 （即使相反的操作没有问题）。 *注意：文件存在于指定路径上的HDFS文件系统上

这是一段代码：

Configuration conf = new Configuration(); conf.set("fs.defaultFS", "${NAMENODE_URI}"); FileSystem hdfsFileSystem = FileSystem.get(conf); String result = ""; Path local = new Path("${SOME_LOCAL_PATH}"); Path hdfs = new Path("${SOME_HDFS_PATH}"); String fileName = hdfs.getName(); if (hdfsFileSystem.exists(hdfs)) { hdfsFileSystem.copyToLocalFile(hdfs, local); result = "File " + fileName + " copied to local machine on location: " + localPath; } else { result = "File " + fileName + " does not exist on HDFS on location: " + localPath; } return result; 

我得到的例外情况如下：

 12/07/13 14:57:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.IOException: Cannot run program "cygpath": CreateProcess error=2, The system cannot find the file specified at java.lang.ProcessBuilder.start(Unknown Source) at org.apache.hadoop.util.Shell.runCommand(Shell.java:206) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.fs.FileUtil$CygPathCommand.(FileUtil.java:412) at org.apache.hadoop.fs.FileUtil.makeShellPath(FileUtil.java:438) at org.apache.hadoop.fs.FileUtil.makeShellPath(FileUtil.java:465) at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:573) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:565) at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:403) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:452) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:774) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:755) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:654) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:259) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:232) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:183) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1837) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1806) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1782) at com.hmeter.hadoop.hdfs.hdfsoperations.HdfsOperations.fileCopyFromHdfsToLocal(HdfsOperations.java:75) at com.hmeter.hadoop.hdfs.hdfsoperations.HdfsOperations.main(HdfsOperations.java:148) Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified at java.lang.ProcessImpl.create(Native Method) at java.lang.ProcessImpl.(Unknown Source) at java.lang.ProcessImpl.start(Unknown Source) ... 22 more 

知道什么可能是一个问题？ 为什么要求cygwin的cyqpath？ 我在Windows 7上运行此代码。

谢谢

Answer 1

尝试使用API​​中的此方法：

 //where delSrc is do you want to delete the source, src and dst you already have and useRawLocalFileSystem should be set to true in your case hdfsFileSystem.copyToLocalFile(delSrc, src, dst, useRawLocalFileSystem); 

在你的情况下，取代：

 hdfsFileSystem.copyToLocalFile(hdfs, local); 

有：

 hdfsFileSystem.copyToLocalFile(false, hdfs, local, true); 

Answer 2

您可以按照以下代码操作：

 public static void main(String args[]){ try { Configuration conf = new Configuration(); conf.set("fs.defaultFS", "hdfs://localhost:54310/user/hadoop/"); FileSystem fs = FileSystem.get(conf); FileStatus[] status = fs.listStatus(new Path("hdfsdirectory")); for(int i=0;i