Skip to main content

Backup & restore

How to backup and restore your Airy Core instance.

Backup#

All the application data, including all the configured sources, channels and templates, are stored in Kafka. As Kafka needs Zookeeper to run properly, the Kafka operational data is stored in Zookeeper. Both Kafka and Zookeeper are deployed as statefulsets in Kubernetes and all the data is stored in PersistentVolumes, provisioned by PersistentVolumeClaims. The location of the persistent volumes is dependent of your implementation. For example, in AWS the Elastic Block Store (EBS) is used to manage the volumes.

To view the PersistentVolumeClaims and PersistentVolumes run:

$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-0-kafka-0 Bound airy-kafka 10Gi RWO gp2 143m
datadir-zookeeper-0 Bound airy-zookeeper 5Gi RWO gp2 119m
datalogdir-zookeeper-0 Bound airy-zookeeper-datalog 5Gi RWO gp2 105m
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
airy-kafka 10Gi RWO Delete Bound default/datadir-0-kafka-0 gp2 155m
airy-zookeeper 5Gi RWO Delete Bound default/datadir-zookeeper-0 gp2 128m
airy-zookeeper-datalog 5Gi RWO Delete Bound default/datalogdir-zookeeper-0 gp2 114m

To Backup all your data, you should make periodical snapshots or backup of the PersistentVolumes.

All the credentials and configurations of the Airy Core instance, is stored in your airy.yaml file.

Restore#

You can restore your Airy Core instance in an empty Kubernetes cluster, by restoring the Kafka and Zookeeper PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

Once you have your volumes restored, either from a backup location or from snapshots, you can create the PVs using the following Kubernetes manifests.

curl -L -s https://raw.githubusercontent.com/airyhq/airy/develop/infrastructure/tools/restore/pv.yaml -O pv.yaml

Modify the file according to your infrastructure configuration by specifying VOLUME_ID, AVAILABILITY_ZONE and REGION.

Apply the Kubernetes manifest to create the PV resources

$ kubectl apply -f ./pv.yaml
persistentvolume/restored-kafka created
persistentvolume/restored-zookeeper created
persistentvolume/restored-zookeeper-datalog created

Run the following command to create the neccessary PVCs:

kubectl apply -f https://raw.githubusercontent.com/airyhq/airy/develop/infrastructure/tools/restore/pvc.yaml
persistentvolumeclaim/datadir-0-kafka-0 created
persistentvolumeclaim/datadir-zookeeper-0 created
persistentvolumeclaim/datalogdir-zookeeper-0 created

Once the storage is restored, transform your airy.yaml file to be compatible with Helm and install the Helm chart for Airy Core, with the desired version:

echo "global:" > helm .yaml
cat airy.yaml | sed 's/^/ /' >> helm.yaml
export AIRY_VERSION="0.30.0"
helm install airy https://airy-core-helm-charts.s3.amazonaws.com/stable/airy-${AIRY_VERSION}-alpha.tgz --values ./helm.yaml --timeout 10m0s

Apply your airy.yaml configuration.

airy config apply
Last updated on