i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'.
My kafka is running but when i want to create a partition.
from kafka import TopicPartition
(ERROR THERE) consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
consumer.assign([TopicPartition('foobar', 2)])
msg = next(consumer)
traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/group.py", line 284, in init self._client = KafkaClient(metrics=self._metrics, **self.config) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 202, in init self.config['api_version'] = self.check_version(timeout=check_timeout) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 791, in check_version raise Errors.NoBrokersAvailable() kafka.errors.NoBrokersAvailable: NoBrokersAvailable
You cannot create partitions within a consumer. Partitions are created when you create a topic. For example, using command line tool:
bin/kafka-topics.sh \
--zookeeper localhost:2181 \
--create --topic myNewTopic \
--partitions 10 \
--replication-factor 3
This creates a new topic "myNewTopic" with 10 partitions (numbered from 0 to 9) and replication factor 3. (see http://docs.confluent.io/3.0.0/kafka/post-deployment.html#admin-operations and https://kafka.apache.org/documentation.html#quickstart_createtopic)
Within your consumer, if you call assign(), it means you want to consume the corresponding partition and this partition must exist already.