项目中需要将本地文件拷贝到hdfs上,由于本人比较懒,于是使用擅长的Java程序通过Hadoop.FileSystem.CopyFromLocalFile方法来实现。 在本地(Window 7 环境)本地模式下运行却遇到了下述异常:
An exception or error caused a run to abort: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:388)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:451)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:430)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1882)
通过分析异常堆栈可知,
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0方法发生了异常,createFileWithMode0方法的实现如下:
/** Wrapper around CreateFile() with security descriptor on Windows */
private static native FileDescriptor createFileWithMode0(String path,
long desiredAccess, long shareMode, long creationDisposition, int mode)
throws NativeIOException;
通过代码可知,这个方法是hadoop不支持的方法。那么为什么会调用这个方法,通过异常堆栈继续向上追踪,
是在 org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init> 过程调用了nativeio.NativeIO$Windows类,对应的方法如下:
1 private LocalFSFileOutputStream(Path f, boolean append,
2 FsPermission permission) throws IOException {
3 File file = pathToFile(f);
4 if (permission == null) {
5 this.fos = new FileOutputStream(file, append);
6 } else {
7 if (Shell.WINDOWS && NativeIO.isAvailable()) {
8 this.fos = NativeIO.Windows.createFileOutputStreamWithMode(file,
9 append, permission.toShort());
10 } else {
11 this.fos = new FileOutputStream(file, append);
12 boolean success = false;
13 try {
14 setPermission(f, permission);
15 success = true;
16 } finally {
17 if (!success) {
18 IOUtils.cleanup(LOG, this.fos);
19 }
20 }
21 }
22 }
23 }
通过调用椎栈可知是上述代码第8行调用了NativeIO.Windows类。那么if判断