Zilla Kafka Proxy Quickstart

Get started with Zilla by deploying our Docker Compose stack. Before proceeding, you should have Docker Compose installed.

Postman Collections

This quickstart uses Aklivity's public Postman Workspace to give you a fast and easy way to try out Zilla's multi-protocol Kafka proxying capabilities.

App or Desktop Agent

Once the collections are forked, you can run them against the local stack if you have either the Postman App or Postman Desktop Agent installed.

Fork each of these collections into your personal/team workspace.

• Zilla - REST Kafka proxy
• Zilla - SSE Kafka proxy
• Zilla - gRPC Kafka proxy
• Zilla - MQTT Kafka proxy

Collection header > View more actions > Create a Fork

Zilla Docker Compose Stack

Download and run the Zilla quickstart. This startup.sh script will start Zilla and everything you need for this quickstart.

wget -qO- https://raw.githubusercontent.com/aklivity/zilla-examples/main/startup.sh | sh -

The key components this script will setup:

• Configured Zilla instance
• Kafka instance and topics
• Kafka UI for browsing topics & messages
• gRPC Route Guide server
• MQTT message simulator

Kafka topics

This Zilla quickstart hosts a UI for the Kafka cluster. To browse the data, go to the topics page.

Find the docker command in the Verify the Kafka topics were created section of the startup.sh script output. Verify these topics are listed:

• items-crud - REST CRUD messages
• events-sse - SSE event messages
• echo-service-messages - gRPC echo messages
• route-guide-requests - gRPC RouteGuide requests
• route-guide-responses - gRPC RouteGuide responses
• iot-messages - MQTT messages responses
• iot-retained - MQTT messages with the retained flag
• iot-sessions - MQTT sessions

REST Kafka proxy

The Zilla REST Kafka proxy can expose common entity CRUD endpoints with the entity data being stored on Kafka topics. Leveraging Kafka's cleanup.policy=compact feature, Zilla enables a standard REST backend architecture with Kafka as the storage layer. Adding an Idempotency-Key header during creation will set the message key and act as the ID for the record. A UUID is generated if no key is sent.

• GET - Fetches all items on the topic or Fetch one item by its key using /:key.
• POST - Create a new item with the Idempotency-Key header setting the key.
• PUT - Update an item based on its key using /:key.
• DELETE - Delete an item based on its key using /:key.

The items-crud Kafka topic will have all the objects you posted, updated, and deleted.

Going Deeper

Zilla can be configured for request-response over Kafka topics both synchronously and asynchronously, and more that we can't cover in this quickstart. Here are some other resources you will want to check out.

• JWT Auth example
• Kafka cache example
• Kafka sync example
• Kafka async example

SSE Kafka proxy

The Zilla SSE Kafka proxy can expose a Kafka topic as a Server-sent Events (SSE) stream, enabling a resilient event-driven architecture to be exposed over HTTP. This quickstart will demonstrate streaming data to one session while posting data from another using Zilla and Kafka as the backend.

• POST - Push a new event.
• GET:SSE - Stream all of the events published on the event-sse Kafka topic.

The events-sse Kafka topic will have a record of each new event sent over HTTP to the SSE stream.

Going Deeper

Zilla can be configured for more use cases we can't cover in this quickstart. Here are some other interesting examples you will want to check out.

• SSE Kafka proxy intro
• Kafka fanout example

gRPC Kafka proxy

The Zilla gRPC Kafka proxy maps the service method's request and response messages directly to Kafka topics. This can include simple RPC request-response messages, but this quickstart demonstrates Simple-RPC, Server-side, Client-side, and Bidirectional streaming RPC to a running gRPC server through the GetFeature, ListFeature, RecordRoute, and RouteChat, respectively. Zilla is routing all of the messages from the client to the server through Kafka. You can match the individual service calls on the topics by the matching key UUIDs which come from the zilla:correlation-id header.

• RouteGuide - Proxy messages through Kafka to a running gRPC server.
• EchoService - Zilla implements a simple message echo service.

Check out the Kafka topics:

The echo-service-messages Kafka topic will have both the request and response record for each of the echo messages sent. You can see the records with the same generated UUIDs and header values.

The route-guide-requests Kafka topic will have every proto request object, meaning every message that is sent to the server. The route-guide-responses Kafka topic will have every proto response object, meaning every message returned from the server.

Going Deeper

Zilla can be configured for more use cases we can't cover in this quickstart. Here are some other interesting examples you will want to check out.

• gRPC proxy guide
• gRPC proxy example
• Kafka fanout example
• Kafka proxy example

MQTT Kafka proxy

The Zilla MQTT Kafka proxy provides an MQTT broker for devices and client libraries to natively interact with Kafka. Clients can connect and send MQTT messages where Zilla will store them in one of three defined Kafka topics. This quickstart manages all messages, messages marked with the retained flag, and sessions on any topic.

Postman MQTT in BETA

Postman recently released MQTT support into public BETA, and we are using it for this quickstart. Be mindful that there may be minor issues encountered using it.

• Pub/Sub - Publish your own messages
• Simulated Topics - Subscribe to simulated traffic

Setting the retain flag to true on your topic will send that message to the retained Kafka topic. After those messages are published, a new subscription will get the last message sent for that topic.

The iot-messages Kafka topic will store every message sent to the broker. The iot-retained Kafka topic will store only messages sent with the retain flag set to true. By log compacting this topic, it will only return the most recent copy of the message to a newly subscribed client. The iot-sessions Kafka topic will have a record for each connection that Zilla has managed between it and the clients. You can see the client-id in the key and the topic in the value when necessary.

An mqtt-simulator is included in the quickstart that will produce mock messages and send them to Zilla. The simulator uses the Python paho-mqtt library and the MQTT v5 specification.

Going Deeper

Zilla can be configured for more use cases we can't cover in this quickstart. Here are some other interesting examples you will want to check out.

• Running an MQTT Kafka broker
• JWT Auth example

Metrics

This Zilla quickstart collects basic metrics for the streaming, HTTP, and gRPC services. Go to http://localhost:7190/metrics to see the Prometheus exported data.