Kafka 환경 구축 하기

OS : Centos

Docker를 통해 Centos 환경을 만든다.

(최초 수행 시) docker run --name centos -it centos /bin/bash
(이후 수행 시) docker exec -it centos /bin/bash


Java 1.8 설치

$ yum install java-1.8.0-openjdk-devel


주키퍼 설치
$ yum install wget
$ wget "https://downloads.apache.org/zookeeper/zookeeper-3.6.0/apache-zookeeper-3.6.0-bin.tar.gz"
$ tar -xvzf apache-zookeeper-3.6.0-bin.tar.gz
$ mkdir -p /usr/local/zookeeper
$ mv apache-zookeeper-3.6.0-bin/* /usr/local/zookeeper
$ mkdir -p /var/lib/zookeeper
$ cat > /usr/local/zookeeper/conf/zoo.cfg << EOF
> tickTime=2000
> dataDir=/var/lib/zookeeper
> clientPort=2181
> EOF
$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
$ /usr/local/zookeeper/bin/zkServer.sh start


설치 확인
$ yum install telnet

$ telnet localhost 2181
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
srvr
Zookeeper version: 3.6.0--b4c89dc7f6083829e18fae6e446907ae0b1f22d7, built on 02/25/2020 14:38 GMT
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: standalone
Node count: 5
Connection closed by foreign host.

> 주키퍼는 카프카와 같은 분산 처리 시스템의 서버들에 관한 메타데이터를 통합 관리 하는데 사용된다.
> 주키퍼의 클러스터를 앙상블이라 한다.
> 앙상블내 주키퍼는 5개로 구성하는 것을 추천한다.

카프카 설치
$ wget "http://apache.mirror.cdnetworks.com/kafka/2.4.1/kafka_2.12-2.4.1.tgz"
$ mkdir -p /usr/local/kafka
$ mv kafka_2.12-2.4.1/* /usr/local/kafka/
$ /usr/local/kafka/bin/kafka-server-start.sh \
> -daemon /usr/local/kafka/config/server.properties

확인
$ /usr/local/kafka/bin/kafka-topics.sh --create \
> --zookeeper localhost:2181 \
> --replication-factor 1 --partitions 1 --topic test
Created topic test.

$ /usr/local/kafka/bin/kafka-console-producer.sh \
> --broker-list localhost:9092 \
> --topic test
>Test Message 1
>Test Message 2
^D

$ /usr/local/kafka/bin/kafka-console-consumer.sh \
> --bootstrap-server localhost:9092 \
> --topic test --from-beginning

Test Message 1
Test Message 2

^CProcessed a total of 2 messages