设为首页 加入收藏

TOP

使用YCSB对HBASE进行测试
2019-03-15 01:48:58 】 浏览:45
Tags:使用 YCSB HBASE 进行 测试

YCSB介绍

YCSB(Yahoo! Cloud Serving Benchmark)是雅虎开源的一款通用的性能测试工具。

通过这个工具我们可以对各类NoSQL产品进行相关的性能测试,包括:

关于YCSB的相关说明可以参考:

  1. Getting Started
  2. Running a Workload
  3. Adding a Database

YCSB与HBase自带的性能测试工具(Performanceeva luation)相比,好处在于:

  • 扩展:进行性能测试的客户端不仅仅只是HBase一款产品,而且可以是HBase不同的版本。
  • 灵活:进行性能测试的时候,可以选择进行测试的方式:read+write,read+scan等,还可以选择不同操作的频度与选取Key的方式。
  • 监控:
    • 进行性能测试的时候,可以实时显示测试进行的进度:
    • 1340sec: 751515operations; 537.74current ops/sec; [INSERT AverageLatency(ms)=1.77]
      1350sec: 755945operations; 442.82current ops/sec; [INSERT AverageLatency(ms)=2.18]
      1360sec: 761545operations; 559.72current ops/sec; [INSERT AverageLatency(ms)=1.71]
      1370sec: 767616operations; 606.92current ops/sec; [INSERT AverageLatency(ms)=1.58]
    • 测试完成之后,会显示整体的测试情况:
    • [OVERALL], RunTime(ms), 1762019.0
      [OVERALL], Throughput(ops/sec), 567.5307700995279
      [INSERT], Operations, 1000000
      [INSERT], AverageLatency(ms), 1.698302
      [INSERT], MinLatency(ms), 0
      [INSERT], MaxLatency(ms), 14048
      [INSERT], 95thPercentileLatency(ms), 2
      [INSERT], 99thPercentileLatency(ms), 3
      [INSERT], Return=0, 1000000
      [INSERT], 0, 29
      [INSERT], 1, 433925
      [INSERT], 2, 549176
      [INSERT], 3, 10324
      [INSERT], 4, 3629
      [INSERT], 5, 1303
      [INSERT], 6, 454
      [INSERT], 7, 140

YCSB的不足在于:

自带的workload模型还是过于简单,不提供MR的形式进行测试,所以进行测试的时候如果希望开启多线程的方式会比较麻烦。

比如导入的时候开启多线程只能是启动多个导入进程,然后在不同的启动参数中指定“开始Key的值”。在进行Transaction测试的时候,只能在多台机器开启多个线程来操作。

使用YCSB测试HBASE-0.90.4

从官网下载YCSB-0.1.3的源代码

http://github.com/brianfrankcooper/YCSB/tarball/0.1.3

编译YCSB

hdfs@hd0004-sw1 guopeng$ cd YCSB-0.1.3/
hdfs@hd0004-sw1 YCSB-0.1.3$ pwd
/home/hdfs/guopeng/YCSB-0.1.3
hdfs@hd0004-sw1 YCSB-0.1.3$ ant
Buildfile: /home/hdfs/guopeng/YCSB-0.1.3/build.xml

compile:
javac /home/hdfs/guopeng/YCSB-0.1.3/build.xml:50: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

makejar:

BUILD SUCCESSFUL
Total time: 0 seconds
hdfs@hd0004-sw1 YCSB-0.1.3$

由于YCSB自带的HBase客户端代码有一些兼容性的问题,所以我们使用如下的代码替换YCSB中自带的文件(db/hbase/src/com/yahoo/ycsb/db/HBaseClient.java):

复制代码
package com.yahoo.ycsb.db;

import java.io.IOException;
import java.util.ConcurrentModificationException;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import java.util.Vector;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;

import com.yahoo.ycsb.DBException;

/**
 * HBase client for YCSB framework
 * @see http://blog.data-works.org
 * @see http://gpcuster.cnblogs.com/
 */
public class HBaseClient extends com.yahoo.ycsb.DB {
    private static final Configuration config = new Configuration();
    
    static {
        config.addResource("hbase-default.xml");
        config.addResource("hbase-site.xml");
    }

    public boolean _debug = false;

    public String _table = "";
    public HTable _hTable = null;
    public String _columnFamily = "";
    public byte _columnFamilyBytes[];

    public static final int Ok = 0;
    public static final int ServerError = -1;
    public static final int HttpError = -2;
    public static final int NoMatchingRecord = -3;

    public static final Object tableLock = new Object();

    /**
     * Initialize any state for this DB. Called once per DB instance; there is
     * one DB instance per client thread.
     */
    public void init() throws DBException {
        if ((getProperties().getProperty("debug") != null)
                && (getProperties().getProperty("debug").compareTo("true") == 0)) {
            _debug = true;
        }

        _columnFamily = getProperties().getProperty("columnfamily");
        if (_columnFamily == null) {
            System.err
                    .println("Error, must specify a columnfamily for HBase table");
            throw new DBException("No columnfamily specified");
        }
        _columnFamilyBytes = Bytes.toBytes(_columnFamily);
        
        // read hbase client settings.
        for (Object key : getProperties().keySet()) {
            String pKey = key.toString();
            if (pKey.startsWith("hbase.")) {
                String pValue = getProperties().getProperty(pKey);
                if (pValue != null) {
                    config.set(pKey, pValue);
                }
            }
        }
    }

    /**
     * Cleanup any state for this DB. Called once per DB instance; there is one
     * DB instance per client thread.
     */
    public void cleanup() throws DBException {
        try {
            if (_hTable != null) {
                _hTable.flushCommits();
            }
        } catch (IOException e) {
            throw new DBException(e);
        }
    }

    public void getHTable(String table) throws IOException {
        synchronized (tableLock) {
            _hTable = new HTable(config, table);
        }

    }

    /**
     * Read a record from the database. Each field/value pair from the result
     * will be stored in a HashMap.
     * 
     * @param table
     *            The name of the table
     * @param key
     *            The record key of the record to read.
     * @param fields
     *            The list of fields to read, or null for all of them
     * @param result
     *            A HashMap of field/value pairs for the result
     * @return Zero on success, a non-zero error code on error
     */
    public int read(String table, String key, Set<String> fields,
            HashMap<String, String> result) {
        // if this is a "new" table, init HTable object. Else, use existing one
        if (!_table.equals(table)) {
            _hTable = null;
            try {
                getHTable(table);
                _table = table;
            } catch (IOException e) {
                System.err.println("Error accessing HBase table: " + e);
                return ServerError;
            }
        }

        Result r = null;
        try {
            if (_debug) {
                System.out.println("Doing read from HBase columnfamily "
                        + _columnFamily);
                System.out.println("Doing read for key: " + key);
            }
            Get g = new Get(Bytes.toBytes(key));
            if (fields == null) {
                g.addFamily(_columnFamilyBytes);
            } else {
                for (String field : fields) {
                    g.addColumn(_columnFamilyBytes, Bytes.toBytes(field));
                }
            }
            r = _hTable.get(g);
        } catch (IOException e) {
            System.err.println("Error doing get: " + e);
            return ServerError;
        } catch (ConcurrentModificationException e) {
            // do nothing for now...need to understand HBase concurrency model
            // better
            return ServerError;
        }

        for (KeyValue kv : r.raw()) {
            result.put(Bytes.toString(kv.getQualifier()),
                    Bytes.toString(kv.getValue()));
            if (_debug) {
                System.out.println("Result for field: "
                        + Bytes.toString(kv.getQualifier()) + " is: "
                        + Bytes.toString(kv.getValue()));
            }

        }
        return Ok;
    }

    /**
     * Perform a range scan for a set of records in the database. Each
     * field/value pair from the result will be stored in a HashMap.
     * 
     * @param table
     *            The name of the table
     * @param startkey
     *            The record key of the first record to read.
     * @param recordcount
     *            The number of records to read
     * @param fields
     *            The list of fields to read, or null for all of them
     * @param result
     *            A Vector of HashMaps, where each HashMap is a set field/value
     *            pairs for one record
     * @return Zero on success, a non-zero error code on error
     */
    public int scan(String table, String startkey, int recordcount,
            Set<String> fields, Vector<HashMap<String, String>> result) {
        // if this is a "new" table, init HTable object. Else, use existing one
        if (!_table.equals(table)) {
            _hTable = null;
            try {
                getHTable(table);
                _table = table;
            } catch (IOException e) {
                System.err.println("Error accessing HBase table: " + e);
                return ServerError;
            }
        }

        Scan s = new Scan(Bytes.toBytes(startkey));
        // HBase has no record limit. Here, assume recordcount is small enough
        // to bring back in one call.
        // We get back recordcount records
        s.setCaching(recordcount);

        // add specified fields or else all fields
        if (fields == null) {
            s.addFamily(_columnFamilyBytes);
        } else {
            for (String field : fields) {
                s.addColumn(_columnFamilyBytes, Bytes.toBytes(field));
            }
        }

        // get results
        ResultScanner scanner = null;
        try {
            scanner = _hTable.getScanner(s);
            int numResults = 0;
            for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
                // get row key
                String key = Bytes.toString(rr.getRow());
                if (_debug) {
                    System.out.println("Got scan result for key: " + key);
                }

                HashMap<String, String> rowResult = new HashMap<String, String>();

                for (KeyValue kv : rr.raw()) {
                    rowResult.put(Bytes.toString(kv.getQualifier()),
                            Bytes.toString(kv.getValue()));
                }
                // add rowResult to result vector
                result.add(rowResult);
                numResults++;
                if (numResults >= recordcount) // if hit recordcount, bail out
                {
                    break;
                }
            } // done with row

        }

        catch (IOException e) {
            if (_debug) {
                System.out
                        .println("Error in getting/parsing scan result: " + e);
            }
            return ServerError;
        }

        finally {
            scanner.close();
        }

        return Ok;
    }

    /**
     * Update a record in the database. Any field/value pairs in the specified
     * values HashMap will be written into the record with the specified record
     * key, overwriting any existing values with the same field name.
     * 
     * @param table
     *            The name of the table
     * @param key
     *            The record key of the record to write
     * @param values
     *            A HashMap of field/value pairs to update in the record
     * @return Zero on success, a non-zero error code on error
     */
    public int update(String table, String key, HashMap<String, String> values) {
        // if this is a "new" table, init HTable object. Else, use existing one
        if (!_table.equals(table)) {
            _hTable = null;
            try {
                getHTable(table);
                _table = table;
            } catch (IOException e) {
                System.err.println("Error accessing HBase table: " + e);
                return ServerError;
            }
        }

        if (_debug) {
            System.out.println("Setting up put for key: " + key);
        }
        Put p = new Put(Bytes.toBytes(key));
        for (Map.Entry<String, String> entry : values.entrySet()) {
            if (_debug) {
                System.out.println("Adding field/value " + entry.getKey() + "/"
                        + entry.getValue() + " to put request");
            }
            p.add(_columnFamilyBytes, Bytes.toBytes(entry.getKey()),
                    Bytes.toBytes(entry.getValue()));
        }

        try {
            _hTable.put(p);
        } catch (IOException e) {
            if (_debug) {
                System.err.println("Error doing put: " + e);
            }
            return ServerError;
        } catch (ConcurrentModificationException e) {
            // do nothing for now...hope this is rare
            return ServerError;
        }

        return Ok;
    }

    /**
     * Insert a record in the database. Any field/value pairs in the specified
     * values HashMap will be written into the record with the specified record
     * key.
     * 
     * @param table
     *            The name of the table
     * @param key
     *            The record key of the record to insert.
     * @param values
     *            A HashMap of field/value pairs to insert in the record
     * @return Zero on success, a non-zero error code on error
     */
    public int insert(String table, String key, HashMap<String, String> values) {
        return update(table, key, values);
    }

    /**
     * Delete a record from the database.
     * 
     * @param table
     *            The name of the table
     * @param key
     *            The record key of the record to delete.
     * @return Zero on success, a non-zero error code on error
     */
    public int delete(String table, String key) {
        // if this is a "new" table, init HTable object. Else, use existing one
        if (!_table.equals(table)) {
            _hTable = null;
            try {
                getHTable(table);
                _table = table;
            } catch (IOException e) {
                System.err.println("Error accessing HBase table: " + e);
                return ServerError;
            }
        }

        if (_debug) {
            System.out.println("Doing delete for key: " + key);
        }

        Delete d = new Delete(Bytes.toBytes(key));
        try {
            _hTable.delete(d);
        } catch (IOException e) {
            if (_debug) {
                System.err.println("Error doing delete: " + e);
            }
            return ServerError;
        }

        return Ok;
    }
}
复制代码

修改后的HBase客户端可以直接在命令行中指定测试需要使用的客户端参数,如zk的连接信息:-p hbase.zookeeper.quorum=hd0004-sw1.dc.sh-wgq.sdo.com,hd0001-sw1.dc.sh- wgq.sdo.com,客户端的本地缓存大小:-p hbase.client.write.buffer=100,等等。

然后拷贝编译和使用依赖的Jar包和配置信息。

[hdfs@hd0004-sw1 YCSB-0.1.3]$ cp ~/hbase-current/*.jar ~/hbase-current/lib/*.jar ~/hbase-current/conf/hbase-*.xml db/hbase/lib/
[hdfs@hd0004-sw1 YCSB-0.1.3]$ 

现在就可以编译HBase的客户端:

[hdfs@hd0004-sw1 YCSB-0.1.3]$ ant dbcompile-hbase
Buildfile: /home/hdfs/guopeng/YCSB-0.1.3/build.xml

compile:
    [javac] /home/hdfs/guopeng/YCSB-0.1.3/build.xml:50: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

makejar:

dbcompile-hbase:

dbcompile:
    [javac] /home/hdfs/guopeng/YCSB-0.1.3/build.xml:63: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

makejar:

BUILD SUCCESSFUL
Total time: 0 seconds
[hdfs@hd0004-sw1 YCSB-0.1.3]$ 

最后要建立测试用的HBase表(usertable):

hbase(main):004:0> create 'usertable', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
0 row(s) in 1.2940 seconds

这样环境就准备好了。

然后使用下面的命令就可以开始导入需要测试的数据了:

java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -load -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p hbase.zookeeper.quorum=hd0004-sw1.dc.sh-wgq.sdo.com,hd0001-sw1.dc.sh-wgq.sdo.com,hd0003-sw1.dc.sh-wgq.sdo.com,hd0149-sw18.dc.sh-wgq.sdo.com,hd0165-sw13.dc.sh-wgq.sdo.com -s


测试:http://www.cnblogs.com/gpcuster/archive/2011/08/16/2141430.html


编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇Memcached、Redis、MongoDB、HBas.. 下一篇HDFS、MR、Kafka、Storm、Spark、..

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(217) }