Airy by default ships with a Kafka cluster that is installed via Helm. If you already have an existing cluster or want more direct control over its setup you can also connect your Airy installation to a remote Kafka cluster. In the following sections we will show you how to either point a running Airy instance. If you want to start a new Airy instance and want to immediately connect it to Kafka have a look here.
This guide does not cover how to migrate your existing data to another Kafka cluster. Instead, all running apps will resume their operation using the data found in the new Kafka cluster. It is therefore also required that all topics in the target cluster are already created.
First up you need to gather the addresses of your Kafka clusters bootstrap servers and add them to your
Since the external Kafka cluster needs to be accessible over the internet it is a good idea to enable authentication. Currently, the only authentication mechanism supported is SASL/PLAIN. You can have a look at this page to learn how to configure your broker to use this method.
This should leave you with a single line JAAS configuration string that you also have to add to your
If you want to also use your own schema registry you can also add it to this config under the
If you do not do this you have to explicitly add the existing schema registry url to this configuration as it will override whatever is currently set. You can look up the current schema registry url by running:
Finally, you can apply this configuration by running the
airy config apply command. Once you have verified that everything is
working you can scale down the default Kafka cluster by running the
kubectl scale sts kafka --replicas 0 command.