Setting up a local Apache Kafka instance for testing

Start into Apache Kafka for a project

This article walks you through that process and hopefully save your some time  to start into Apache Kafka.

Installing the binaries

There’s a quickstart guide over at the official docs.

The Docker way

This is by far the easiest we have found and saves you ton of time. Once you have cloned the repo just spin it up!

docker-compose up -d

If you want to know what’s going on in here let’s take a look at the docker-compose.yaml file.

docker-compose.yaml 

---
version: '3'
services:
 zookeeper:
 image: confluentinc/cp-zookeeper:7.0.1
 container_name: zookeeper
 environment:
 ZOOKEEPER_CLIENT_PORT: 2181
 ZOOKEEPER_TICK_TIME: 2000

 broker:
 image: confluentinc/cp-kafka:7.0.1
 container_name: broker
 ports:
 # To learn about configuring Kafka for access across networks see
 # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
 - "9092:9092"
 depends_on:
 - zookeeper
 environment:
 KAFKA_BROKER_ID: 1
 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

1. We spin up a Zookeeper instance at port2181 internally within the Docker network (i.e. not accessible from the host)
2. Next we spin up our broker which is the Kafka instance which has a dependency on Zookeeper. This is exposed at port 9092 and accessible from the host.

Everything else is set to default. If you want to learn what each environment variable does, head over here . You can add multiple brokers (Kafka instances) if you want in the same docker-compose file.

This approach is quite beneficial if you want to stand up an instance for testing on the go or even in a CI environment.

Interacting with the cluster

Before we publish anything, let’s spin up a consumer so that we know our setup is working. To do that, run the following command from the CLI.

docker exec --interactive --tty broker \
kafka-console-consumer --bootstrap-server localhost:9092 \
 --topic example-topic \
 --from-beginning

Now let’s create some messages and see them in action!

docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server localhost:9092 \
 --topic example-topic

This is what it will look like from the terminal. To your left is theproducerandconsumerto your right hand side.

A GUI perhaps?

Not a fan of the CLI? There’s many tools out there. Lightweight solution for this you decided to go ahead with Tools for Apache Kafka for VS Code .

Once you have that going you could interact with it from the side bar. There will be a new icon with Kafka’s logo on it!

Here’s what it looks like when it finds your instance.

producer.kafka 

PRODUCER keyed-message
topic: example-topic
key: mykeyq
record content

###

PRODUCER non-keyed-json-message
topic: example-topic
{
 "type": "my_test_event-{}"
}

consumer.kafka 

CONSUMER consumer-group-id
topic: example-topic
partitions: 0
from: 1

You can invoke the producers and consumers right within VS Code, which is pretty easy and cool!

References

1. https://developer.confluent.io/quickstart/kafka-docker/ 
2. https://marketplace.visualstudio.com/items?itemName=jeppeandersen.vscode-kafka 
3. https://docs.confluent.io/platform/current/kafka/multi-node.html