版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/JavaMrZhang/article/details/79777573
sequence文件存储格式
- txt
- 纯文本格式,若干行记录。默认用字符编码存储
- SequenceFile格式(顺序文件格式,可进行切割)
- key-value 格式进行存储,最终形成的是一个二进制文件, 需用hadoop提供的api进行写入存储。
- 编写 写入 seq文件案例。
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS","hdfs://s100:8020");
FileSystem fileSystem = FileSystem.get(configuration);
Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
SequenceFile.Writer writer = SequenceFile.createWriter(fileSystem, configuration, path, IntWritable.class, Text.class);
writer.append(new IntWritable(1),new Text("gg1"));
writer.append(new IntWritable(1),new Text("gg2"));
writer.append(new IntWritable(1),new Text("gg3"));
writer.append(new IntWritable(1),new Text("gg4"));
writer.close();
- 编写读取 seq 文件案例
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS","hdfs://s100:8020");
FileSystem fileSystem = FileSystem.get(configuration);
Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
SequenceFile.Reader sr = new SequenceFile.Reader(fileSystem,path,configuration);
IntWritable key = new IntWritable();
Text value = new Text();
while (sr.next(key,value)){
System.out.println(key +":"+value );
}
- 查看文件内容
$> hdfs dfs -text /user/myfile.seq
$> hdfs dfs -cat /user/myfile.seq (此命令查看会出现乱码)
seq 文件格式解析
顺序文件由文件头和随后的一条或多条记录组成
---文件头------
--key-value----sync
--key-value----
--key-value----
--key-value----
--key-value----sync
--key-value----
--key-value----
--key-value----sync
- 文件头格式
- SEQ+版本号+key类型class+value类型class + 压缩格式类型
- 代码案例
public void seekSeq() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS","hdfs://s100:8020");
FileSystem fileSystem = FileSystem.get(configuration);
Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
SequenceFile.Reader sr = new SequenceFile.Reader(fileSystem,path,configuration);
IntWritable key = new IntWritable();
Text value = new Text();
sr.seek(253);
sr.next(key,value);
System.out.println(key +" : " + value);
sr.close();
}
public void sync() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS","hdfs://s100:8020");
FileSystem fileSystem = FileSystem.get(configuration);
Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
SequenceFile.Reader sr = new SequenceFile.Reader(fileSystem,path,configuration);
IntWritable key = new IntWritable();
Text value = new Text();
int syncPos = 12;
sr.sync(syncPos);
long pos = sr.getPosition();
sr.next(key,value);
System.out.println("syncPos : " + syncPos + "pos : " + pos +"key : "+key+"value : " + value);
}
MapFile文件格式
- 是排序的seqfie,具有索引。要求key按照大小顺序添加
- 包含两个文件
- index 文件:索引和偏移量的映射,可以设置间隔,默认128(解释:第128key位置--->第256字节 指存入128个key 对应的 第128key的末尾位置是第128字节的位置。)
- data 文件:存放真实的数据。格式为key -value 。和seqfile文件类似
具体的API可参看Hadoop权威指南第四章 4.5 基于文件的数据结构