Quantcast
Channel: Kafka Timeline
Viewing all articles
Browse latest Browse all 1519

Consumer Parallelism

$
0
0
Hi,

We are using the following method on ConsumerConnector to get multiple
streams per topic, and we have multiple partitions per topic. It looks like
only one of the runnable is active through a relative long time period. Is
there anything we could possible missed?

public <K,V> Map<String, List<KafkaStream<K,V>>>
createMessageStreams(Map<String, Integer> topicCountMap, Decoder<K>
keyDecoder, Decoder<V> valueDecoder);

Then we loop through the streams and create multiple runnable to consumer
the data.

for (KafkaStream<Object, Object> kafkaStream : streams) {
ConsumerRunnable consumerRunnable = runnableBuilder.get()
.setMessageConsumer(consumer)
.setKafkaStream(kafkaStream)
.build();

executor.submit(consumerRunnable);

Best Regards,
Mingtao

Viewing all articles
Browse latest Browse all 1519

Trending Articles