Back atom.xml blog Publishing to Azure Event Hubs from Rust 04 Apr 2019 rust kafka azure

Turn back now, this blog post is so niche that it’s statistically impossible for you to find this useful. Last night I was thinking about building a little app which needed to deal with an event stream, and started poking around the Azure Event Hubs documentation. I noticed that they apparently can now speak Kafka which means I can use my existing Kafka library tooling, nice! Since I was already working with Kafka and Rust for another little project, I took a quick detour and tried to see if I could publish to an Event Hub over Kafka, from Rust. As luck would have it, I can!

The one quirk with Azure Event Hubs compared to how most people use Kafka was the SASL/TLS authentication and encryption, was a bit tricky to use with the rdkafka crate, which built on top of the fantastic librdkafka. SASL/TLS is seems to have become the de facto standard for running Kafka-as-a-Service like AWS MSK or Confluent Cloud, so it is wonderful to figure out how to use the rdkafka crate in this environment.

The relevant code from my gist which demonstrates this functionality is the ClientConfig for the producer:

/*
* NOTE: connection is the full connection string for the Event Hub
*/
let producer: FutureProducer = ClientConfig::new()
.set("bootstrap.servers", &brokers)
.set("produce.offset.report", "true")
.set("message.timeout.ms", "5000")
.set("security.protocol", "sasl_ssl")
/*
* The following setting -must- be set for librdkafka, but doesn't seem to do anything
* worthwhile,. Since we're not relying on kerberos, let's just give it some junk :)
*/
.set("sasl.kerberos.kinit.cmd", "echo 'who wants kerberos?!'")
/*
* Another unused setting which is required to be present
*/
.set("sasl.kerberos.keytab", "keytab")
.set("sasl.mechanisms", "PLAIN")
/*
* The username that Azure Event Hubs uses for Kafka is really this
*/
/*
* NOTE: Depending on your system, you may need to change this to adifferent location
*/
.set("ssl.ca.location", "/etc/ssl/ca-bundle.pem")