设为首页 加入收藏

TOP

复制本地文件到HDFS本地测试异常(一)
2017-10-10 12:10:49 】 浏览:8751
Tags:复制 本地 文件 HDFS 测试 异常

项目中需要将本地文件拷贝到hdfs上,由于本人比较懒,于是使用擅长的Java程序通过Hadoop.FileSystem.CopyFromLocalFile方法来实现。 在本地(Window 7 环境)本地模式下运行却遇到了下述异常:

An exception or error caused a run to abort: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor; 
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
    at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:388)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:451)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:430)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
    at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1882)

通过分析异常堆栈可知,

org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0方法发生了异常,createFileWithMode0方法的实现如下:
    /** Wrapper around CreateFile() with security descriptor on Windows */
    private static native FileDescriptor createFileWithMode0(String path,
        long desiredAccess, long shareMode, long creationDisposition, int mode)
        throws NativeIOException;

通过代码可知,这个方法是hadoop不支持的方法。那么为什么会调用这个方法,通过异常堆栈继续向上追踪,

是在 org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init> 过程调用了nativeio.NativeIO$Windows类,对应的方法如下:

 1  private LocalFSFileOutputStream(Path f, boolean append,
 2         FsPermission permission) throws IOException {
 3       File file = pathToFile(f);
 4       if (permission == null) {
 5         this.fos = new FileOutputStream(file, append);
 6       } else {
 7         if (Shell.WINDOWS && NativeIO.isAvailable()) {
 8           this.fos = NativeIO.Windows.createFileOutputStreamWithMode(file,
 9               append, permission.toShort());
10         } else {
11           this.fos = new FileOutputStream(file, append);
12           boolean success = false;
13           try {
14             setPermission(f, permission);
15             success = true;
16           } finally {
17             if (!success) {
18               IOUtils.cleanup(LOG, this.fos);
19             }
20           }
21         }
22       }
23     }

通过调用椎栈可知是上述代码第8行调用了NativeIO.Windows类。那么if判断

首页 上一页 1 2 下一页 尾页 1/2/2
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇scala学习之第二天:可变容器与不.. 下一篇play for scala 在模板中格式化Da..

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目