Setting Up Kafka cluster on Ubuntu Platform

Setting Up Kafka cluster on Ubuntu Platform
Written by Abhishek JalanApril 25, 2020
3 min read
Kafka
7 VIEWS 1 LIKES 0 DISLIKES SHARE
1 LIKES 0 DISLIKES 7 VIEWS SHARE
Abhishek Jalan

DevSecOps Engineer

In this blog, I will guide you through the process of setting up your own 3-broker Kafka cluster using the Confluent Community.

How to setup 3-broker Kafka cluster using Confluent Community on Ubuntu Platform

Prerequisite:

  • Deploy or get ready with 3 Ubuntu Servers (Min Req. 1CPU/2GB/30GB HDD)
  • Change the hostname and ssh with root credentials to all 3 nodes

for eg:

IP Address Hostname

192.168.0.10 broker1

192.168.0.20 broker2

192.168.0.30 broker3

SSH to all 3 nodes and follow the below steps:

On all three nodes, add the GPG key and package repository, then install Confluent and Java.

 wget -qO - https://packages.confluent.io/deb/5.2/archive.key | sudo apt-key add -
 sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/5.2 stable main"
 sudo apt-get update && sudo apt-get install -y openjdk-8-jdk confluent-community-2.12

On all three nodes, edit the hosts file.

sudo vi /etc/hosts

Add the following entries to the hosts file on all three servers. Use the private IP addresses of your three servers (you can find them in Cloud Playground).

192.168.0.10 zoo1
192.168.0.20 zoo2
192.168.0.30 zoo3

Edit the ZooKeeper config file.

 sudo vi /etc/kafka/zookeeper.properties

Delete the contents of the config file, and replace them with the following:

 tickTime=2000
 dataDir=/var/lib/zookeeper/
 clientPort=2181
 initLimit=5
 syncLimit=2
 server.1=zoo1:2888:3888
 server.2=zoo2:2888:3888
 server.3=zoo3:2888:3888
 autopurge.snapRetainCount=3
 autopurge.purgeInterval=24

Set the Zookeeper ID for each server.

 sudo vi /var/lib/zookeeper/myid

On each server, set the contents of /var/lib/zookeeper/myid to the server's ID. On broker1, enter 1. On broker2, enter 2, and on broker3, enter 3.

 <server id 1, 2, or 3>

Edit the Kafka config file.

 sudo vi /etc/kafka/server.properties

Edit the broker.id, advertised.listeners, and zookeeper.connect in the config file. Set the broker ID to the appropriate ID for each server (1 on broker1, 2 on broker2, and 3 on broker3 ).

For advertised.listeners, provide the hostname for each server: zoo1, zoo2, or zoo3 as appropriate.

Set zookeeper.connect to zoo1:2181.

broker.id=<server id 1, 2, or 3>
advertised.listeners=PLAINTEXT://<hostname zoo1, zoo2, or zoo3>:9092
zookeeper.connect=zoo1:2181

Start and enable the Zookeeper service.

sudo systemctl start confluent-zookeeper
sudo systemctl enable confluent-zookeeper

Wait a few seconds, then do the same for the Kafka service.

sudo systemctl start confluent-kafka
sudo systemctl enable confluent-kafka

Check the services to make sure they are running. Both services should be active (running) on all three servers.

sudo systemctl status confluent*

Test your cluster by listing the current topics.

kafka-topics --list --bootstrap-server localhost:9092

Since you have not created any topics yet, you will only see a default topic. The output should look like this:

__confluent.support.metrics

kafka
ubuntu
7 VIEWS 1 LIKES 0 DISLIKES SHARE
1 LIKES 0 DISLIKES 7 VIEWS SHARE
Was this blog helpful?
You must be Logged in to comment
Code Block
Abhishek Jalan
DevSecOps Engineer
+21 more
17 Blog Posts
4 Discussion Threads
Trending Categories
93
Software12
DevOps34
Frontend Development13
Backend Development13
Server Administration13
Linux Administration12
Data Center14
Sentry11
Terraform15
Ansible9
Docker13
Penetration Testing12
Kubernetes12
NGINX8
JenkinsX10
Jenkins18
SSL6
Ethical-Hacking10
Python8
NodeJs9
RedHat8
Github11
AngularJs15
Google Cloud Platform (GCP)6
SonarQube9
Amazon Web Service (AWS)2
VMware17
Blockchain6
Snipe-IT7