This guide helps you to understand how to install Apache Kafka on Windows 10 operating system and executing some of the basic commands on Kafka console.
Apache Kafka:
Apache Kafka is a distributed, fast and scalable messaging queue platform, which is capable of publishing and subscribing to streams of records, similar to a message queue or enterprise messaging system.
Prerequisites:
Install JRE before you further.
Install Apache Kafka on Windows:
Download the latest Apache Kafka from the official Apache website for me it is 2.11.2.0.0 release.
Click on above highlighted binary downloads and it will be redirected to Apache Foundations main downloads page like below.
Select the above-mentioned apache mirror to download Kafka, it will be downloaded as a .tgz. Extract it and you will see the below folder structure.
/bin directory represents all the binary files which are helpful to start Kafka server different operating systems. As we are working with the windows machine, there will be a folder named windows under /bin directory, which has all the windows related stuff.
/config directory contains all configuration details about Kafka server, zookeeper, and logs. All configurations have their default values if you wanted to change any config details like port you can freely go and change accordingly.
# The directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0server.properties
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs/libs directory contain all dependency executables like java, jetty, log4j and etc.
If you are okay with the above default configurations, you are ready to start zookeeper and Kafka server.
Start the Zookeeper:
Go to Kafka_X.XX-X.X.X\bin\windows\
D:\Softwares\kafka_2.11-2.0.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties
[2018-11-17 03:20:58,713] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-11-17 03:20:58,713] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-11-17 03:20:58,744] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-11-17 03:20:58,744] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-11-17 03:21:03,261] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
.............
.............
[2018-11-17 03:21:03,345] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2018-11-17 03:21:03,345] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-17 03:21:35,434] INFO Accepted socket connection from /127.0.0.1:63924 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-17 03:21:35,450] INFO Client attempting to establish new session at /127.0.0.1:63924 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-11-17 03:21:35,450] INFO Creating new log file: log.5 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2018-11-17 03:21:35,535] INFO Established session 0x1000238240f0000 with negotiated timeout 6000 for client /127.0.0.1:63924 (org.apache.zookeeper.server.ZooKeeperServer)Start Apache Kafka:
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-server-start.bat ../../config/server.properties
[2018-11-17 03:21:30,291] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-11-17 03:21:30,777] INFO starting (kafka.server.KafkaServer)
[2018-11-17 03:21:30,777] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2018-11-17 03:21:30,831] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2018-11-17 03:21:35,350] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:host.name=DESKTOP-RN4SMHT (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.version=9.0.4 (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.home=C:\Program Files\Java\jdk-9.0.4 (org.apache.zookeeper.ZooKeeper)
................
................
[2018-11-17 03:21:43,100] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2018-11-17 03:21:43,138] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-11-17 03:21:43,154] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-11-17 03:21:43,154] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2018-11-17 03:21:43,200] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2018-11-17 03:21:43,235] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2018-11-17 03:21:43,238] INFO Kafka version : 2.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-11-17 03:21:43,238] INFO Kafka commitId : 3402a8361b734732 (org.apache.kafka.common.utils.AppInfoParser)
[2018-11-17 03:21:43,238] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)Note: Should start the zookeeper prior to Kafka server.
You can see the logs, we have successfully started the Zookeeper and Kafka servers.
Then why can’t we create a simple topic and try to exchange some messages between producer and consumer let’s start to do that
Create a Kafka Topic:
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic items-topic
Created topic "items-topic".Creating Kafka Producer:
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic items-topic
>hai chandra
>helloCreating Kafka Consumer:
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic items-topic --from-beginning
hai chandra
helloIf you see these messages on consumer console, you all done. Then you can play with producer and consumer terminal bypassing some Kafka messages.
References:
Happy Learning 🙂
