Multiplexer consumer and producer at kafka

存储架构 Hello, buddy! (源链)

In my kafka consumer threads(high level), after I consumed a message I am applying some business logic to this message and forwarding this to a WS. But this webservice may be down sometimes and since I consumed this object from kafka and offset is moved forward, i would missed this object.

One way get rid of from this problem is to disabling autocommit in zookeeper and committing offset by calling programmaticaly but i expect that this is a very costly operation. I will be producing to kafka at about 2000 tps and may increase later times.

Another way – which i am not sure if it is a good idea – is if i face with any problem, producing this consumed object to kafka again but i didn’t see any post related to this across all my googleings. Is this a thing which is even not considerable?

Can you please give me some insights about handling this situation.


You can post back the failed message to the same topic or another of your choice.

If you use the same topic, you will push the messages at the end of the topic and they will be picked up after the others (so if order matters to you don’t do this). Also if the action that you perform before sending the message is not idempotent you will have to something to identifying this records so they don’t perform the action twice.

If you use a failed_topic, you can push the messages that you can’t send to this topic and when the WS is healthy again you need to create a consumer that consumes all the messages there and sends them to the WS.

Hope it helps!


Architecting and building end-to-end streaming app... In this episode of theData Show, I spoke with Karthik Ramasamy , adjunct faculty member at UC Berkeley, former engineering manager at Twitter, ...
Confluent Partnership Here at Rittman Mead, we are continually broadening the scope and expertise of our services to help our customers keep pace with today's ever-cha...
LinkedIn Announces Open Source Tool to Keep Kafka ... Today at The Kafka Summit in San Francisco, LinkedIn announced a new load balancing tool called Cruise Control, which has been developed to he...
Spring Kafka 1.3.3, 2.0.3, 2.1.2 发布,升级部分 Spring 依赖... Spring Kafka 为三个分支发布了新版本,分别是 1.3.3, 2.0.3 及 2.1.2。具体更新如下: Spring Kafka 1.3.3 9c255d6 Upgrade to Spring Framework 4.3.14, Retry 1.2.2 52b...
Kafka Manager kafka manager作为一款在kafka出世不久就被贡献出来的开源kakfa管理软件,相信使用者不在少数,不过这款软件更新总是有点慢,而且缺少安全认证模块,不过鉴于很多中小公司没人力自研kafka管理平台,这款软件还是可以凑合使用的。由于之前公司的kafka早已升级至0.10.0版本,而早起的...
Hello, buddy!责编内容来自:Hello, buddy! (源链) | 更多关于

本站遵循[CC BY-NC-SA 4.0]。如您有版权、意见投诉等问题,请通过eMail联系我们处理。
酷辣虫 » Multiplexer consumer and producer at kafka

专业 x 专注 x 聚合 x 分享 CC BY-NC-SA 4.0

使用声明 | 英豪名录