Deploying Kafka message queue on Ubuntu machine

Posted on

Deploying Kafka message queue on Ubuntu machine

step

  1. Installation depends on zookeeper
  2. Install Kafka
  3. Configure Kafka
  4. Start and stop Kafka
  5. Test Kafka

1 dependency: install zookeeper

apt-get update
apt-get install zookeeper

The configuration file is in the/etc/zookeeper/confUnder the directory

If it is a stand-alone machine, no additional configuration is required. If there are multiple ZKS, please configure themserver.nAddress.

# ls
configuration.xsl  environment  log4j.properties  myid  zoo.cfg

2 installation of Kafka

mkdir kafka
cd kafka
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz  

tar -xzf kafka_2.12-0.10.2.0.tgz --strip 1

Unzip to the current directory

3 configure Kafka

Configuration file path~/kafka/config/server.properties

By default, Kafka does not allow you to delete topics. To be able to delete topics, add the following line at the end of the file:

delete.topic.enable = true

Remote access requires a host name

Advertised. Host. Name = external IP address

And zookeeper’s address

Zookeeper. Connect = zookeeper external IP address: 2181

4 start and stop Kafka

Start Kafka

Configure startup scriptsstart_kafka.shThe main purpose of using the startup script is to make it run in the background

#!/bin/bash
echo "Begin to start kafka"
nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &
echo "......"

Set permissions

chmod a+x start_kafka.sh
Start Kafka
./start_kafka.sh

Can be in~/kafka/kafka.logYou can see the startup log output in the log file

Stop Kafka

~/kafka/bin/kafka-server-stop.sh

5 test Kafka

Publish and consume a “Hello world” message to ensure the normal operation of Kafka server.

Using two different terminals

Consumer terminal

~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic helloTopic --from-beginning

Publishing terminal

echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic helloTopic > /dev/null

Publish the output of the terminal

$ echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic helloTopic > /dev/null
[2017-12-10 21:51:14,053] WARN Error while fetching metadata with correlation id 1 : {helloTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

No,helloTopicThis topic will automatically create one.

Output of consumption terminal

$ ~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic helloTopic --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello, World

Got a “Hello, world” message.

This means that the server is configured.

If you need remote access, pay attention to the settingsadvertised.host.nameandzookeeper.connect, use the external IP address.

Leave a Reply

Your email address will not be published.