设为首页 加入收藏

TOP

使用Python开发Kafka消息生产者
2019-02-12 02:30:50 】 浏览:14
Tags:使用 Python 开发 Kafka 消息 生产者
版权声明:本文为博主原创文章,未经博主允许不得转载。如有问题,欢迎指正! https://blog.csdn.net/sun7545526/article/details/44624825

安装Kafka-Python


下载Kafka-Python压缩包
https://github.com/mumrah/kafka-python/releases中下载最新的release包,此时最新的包为kafka-python-0.9.3.tar.gz

编译安装
$tar -xvf kafka-python-0.9.3.tar.gz
$cdkafka-python-0.9.3.tar.gz
$python setup.py install

如果报下面的异常说明没有安装setuptools,请参考下文进行安装:
[root@hadoop01 kafka-python-0.9.3]# python setup.py  install
Traceback (most recent call last):
  File "setup.py", line 3, in <module>
    from setuptools import setup, Command
ImportError: No module named setuptools



编码

import time
__author__ = 'aihua.sun'
import logging
import random,string
from kafka.producer import SimpleProducer
from kafka.client import KafkaClient
LOG = logging.getLogger('kafka_producer')
class TrueCloudDataPointProducer():
    def __init__(self,hosts,batch_send=False,batch_send_every_n=20,topic="true_cloud_datapoint_topic"):
        self.hosts=hosts
        self.client=KafkaClient(self.hosts)
        self.batch_send=batch_send
        self.batch_send_every_n=batch_send_every_n
        self.producer = SimpleProducer(self.client,batch_send=batch_send,batch_send_every_n=batch_send_every_n)
        self.topic=topic
    def send_messages(self,msg):
        self.producer.send_messages(self.topic,msg)
def get_instance():
    hosts={'hadoop01':'9092','hadoop01':'9093','hadoop01':'9094','hadoop101':'9095','hadoop02':'9092','hadoop02':'9093','hadoop02':'9094'}
    return TrueCloudDataPointProducer(hosts)
if __name__=="__main__":
    begin=time.time()
    producer=get_instance()
    for i in range(0,10000):
        msg='Message'+str(i)+' '+''.join(random.choice(string.lowercase) for i in range(64))+'\n'
        producer.send_messages(msg)
    end=time.time()
    print("use time:"+str((end-begin)))



安装依赖模块

setuptools

下载

编译安装
$tar zxvf setuptools-14.3.1.tar.gz
$cd setuptools-14.3.1
$python setup.py build
$
python setup.py install

six

编译安装
$tar zxvfsix-1.9.0.tar.gz
$cd setuptools-0.6c11
$python setup.py build
$
python setup.py install









编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇kafka-0.8.1.1总结 下一篇Mac使用IDEA搭建kafka开发环境

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(214) }