Deploying Kafka message queue on Ubuntu machine

step

1. Installation depends on zookeeper
2. Install Kafka
3. Configure Kafka
4. Start and stop Kafka
5. Test Kafka

1 dependency: install zookeeper

apt-get update
apt-get install zookeeper

The configuration file is in the/etc/zookeeper/confUnder the directory

If it is a stand-alone machine, no additional configuration is required. If there are multiple ZKS, please configure themserver.nAddress.

# ls
configuration.xsl  environment  log4j.properties  myid  zoo.cfg

2 installation of Kafka

mkdir kafka
cd kafka
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz  

tar -xzf kafka_2.12-0.10.2.0.tgz --strip 1

Unzip to the current directory

3 configure Kafka

Configuration file path~/kafka/config/server.properties

By default, Kafka does not allow you to delete topics. To be able to delete topics, add the following line at the end of the file:

delete.topic.enable = true

Remote access requires a host name

Advertised. Host. Name = external IP address

And zookeeper’s address

Zookeeper. Connect = zookeeper external IP address: 2181

4 start and stop Kafka

Start Kafka

Configure startup scriptsstart_kafka.shThe main purpose of using the startup script is to make it run in the background

#!/bin/bash
echo "Begin to start kafka"
nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &
echo "......"

Set permissions

chmod a+x start_kafka.sh

Start Kafka

./start_kafka.sh

Can be in~/kafka/kafka.logYou can see the startup log output in the log file

Stop Kafka

~/kafka/bin/kafka-server-stop.sh

5 test Kafka

Publish and consume a “Hello world” message to ensure the normal operation of Kafka server.

Using two different terminals

Consumer terminal

~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic helloTopic --from-beginning

Publishing terminal

echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic helloTopic > /dev/null

Publish the output of the terminal

$ echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic helloTopic > /dev/null
[2017-12-10 21:51:14,053] WARN Error while fetching metadata with correlation id 1 : {helloTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

No,helloTopicThis topic will automatically create one.

Output of consumption terminal

$ ~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic helloTopic --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello, World

Got a “Hello, world” message.

This means that the server is configured.

If you need remote access, pay attention to the settingsadvertised.host.nameandzookeeper.connect, use the external IP address.