In this tutorial, we are going to see how to install Apache Kafka on Ubuntu 18.04 Operating System.

Apache Kafka on Ubuntu:

Apache Kafka is a distributed, fast and scalable messaging queue platform, which is capable of publishing and subscribing to streams of records, similar to a message queue or enterprise messaging system.

1. Prerequisites:

Apache Kafka required Java to run. Before moving further this article, we recommend installing Java on your machine if you don’t have already.

1.1 Update packages:

Before installing any package, it is better to update your ubuntu packages.

$ sudo apt update
[sudo] password for cgoka: 
Hit:1 http://archive.canonical.com bionic InRelease
Hit:2 http://ppa.launchpad.net/linuxuprising/java/ubuntu bionic InRelease                                                                      
Hit:3 https://download.docker.com/linux/ubuntu bionic InRelease                                                                                
....
.....

2. Install Apache Kafka on Ubuntu:

Follow the below steps to install Apache Kafka.

2.1 Download Apache Kafka

Download Apache Kafka binary from the official website. Below wget command helps you to download the Kafka 2.11 version from Apache distributions.

$ wget http://www-us.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

2.2 Extract the downloaded file

tar xzf kafka_2.11-2.0.0.tgz

2.3 Move the folder to user/local directory

Note: It needs sudo access to create a folder inside the user directory. If you have sudo access change to sudo user.

mv kafka_2.11-2.0.0.tgz /usr/local/kafka

3. Start Kafka Server

Kafka uses zookeeper to run, so we have to start zookeeper server prior to the Kafka server.

3.1 Start Zookeeper server:

$ cd /usr/local/kafka
root@work:/usr/local/kafka/bin# ./zookeeper-server-start.sh ../config/zookeeper.properties 
[2019-03-25 23:04:09,468] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-03-25 23:04:09,472] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-03-25 23:04:09,472] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-03-25 23:04:09,472] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
.....
.....

Now we are ready to start the Kafka server.

3.3 Start Kafka Server:

root@work:/usr/local/kafka/bin# ./kafka-server-start.sh ../config/server.properties 
[2019-03-25 23:04:58,072] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-03-25 23:04:58,556] INFO starting (kafka.server.KafkaServer)
[2019-03-25 23:04:58,557] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-03-25 23:04:58,595] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-03-25 23:04:58,602] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)

4. Create a Kafka Topic:

Let’s create a Kafka topic named items-topic with a single partition and one replica.

cgoka@work:/usr/local/kafka/bin$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-topicCreated topic "hello-topic".

The replication factor tells how many copies fo the data will be created, as we are running with single instance just keep it as 1. The partitions option helps us to provide the number of brokers which we want to split between.

5. Create a Kafka Producer:

Kafka producer is a process, which is responsible to put data into Kafka.  The Kafka provides a command line client which will take the input from a file or from standard input and send it out to messages to the Kafka cluster.

Let’s create a Kafka producer to send some messages to the Kafka server.

cgoka@work:/usr/local/kafka/bin$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
>Hello Kafka

6. Create a Kafka Consumer:

Kafka also provides a command line interface for the consumer to read data from the cluster.

cgoka@work:/usr/local/kafka/bin$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello-topic --from-beginning
Hello Kafka

This consumer displays the data as soon as the publisher sends messages to the cluster.

References:

Apache Kafka quick start

Happy Learning 🙂