Trigger functions from Kafka

Trigger function invocations from messages received on Kafka topics.

Note: This feature is included for OpenFaaS Standard & For Enterprises customers.

Read a tutorial on the blog:

• Trigger your functions from Kafka with Confluent Cloud
• Event-driven OpenFaaS with Managed Kafka from Aiven

Installation

You can install the Kafka connector using its helm chart, or by using arkade.

Installation with Helm

See helm chart

Installation with arkade

export TOPICS="payment.created"
arkade install kafka-connector \
 --broker-hosts kafka-broker:9092 \
 --topics $TOPICS \
 --license-file $HOME/.openfaas/LICENSE

Usage

The topics you need to connect to functions are set on a connector.

A single connector can have either a single topic, or a comma-separated list of topics.

Customers tend to prefer to deploy a single copy of the connector for each topic, so that they can be scaled to match the number of consumers set on the partition.

But it's also possible to use a single connector and pass in multiple topics i.e. payment.created,customer.onboarded,invoice.generate.

Your function(s) can then subscribe to one or more topics by setting the topic annotation with the name of the topic, or a comma separated list of topics.

Create a new function:

export OPENFAAS_PREFIX=ghcr.io/openfaas
faas-cli new --lang go provision-customer

Now add an annotation for the payment.created topic, so that the provision-customer function is invoked for any message received:

version: 1.0
provider:
  name: openfaas
  gateway: http://127.0.0.1:8080

functions:
  provision-customer:
    annotations:
      topic: payment.created
    lang: go
    handler: ./provision-customer
    image: ghcr.io/openfaas:provision-customer

Now deploy your function, and publish an event to your Kafka broker on the payment.created topic.

Message body, headers and metadata

The body of the message in binary format will be received as the body of the function or incoming HTTP request.

For headers and metadata:

• X-Topic - the Kafka topic
• X-Kafka-* - each header on the Kafka message is outputted with the form: X-Kafka-Key: Value
• X-Kafka-Partition - the partition in Kafka that the message was received on
• X-Kafka-Offset - the offset in Kafka that the message was received on
• X-Kafka-Key - if set on the message, the key of the message in Kafka

The default content-type is configured as text/plain, but can be changed to another content-type such as application/json or application/octet-stream by the values.yaml file for the connector.

Most templates make these variables available through their request or context object, for example:

• golang-middleware
• python3-http
• node17

For detailed examples with Node.js, see: Serverless For Everyone Else

For detailed examples with Go, see: Everyday Golang (Premium Edition)

See also

• Event-driven OpenFaaS with Managed Kafka from Aiven
• Quick start with the Helm chart - self-hosted, SASL and TLS Client Certificates
• Launch blog post: Staying on topic: trigger your OpenFaaS functions with Apache Kafka

Would you like a demo?

Feel free to reach out to us for a demo or to ask any questions you may have.

• Talk to us