Install GlusterFS

GlusterFS is an in-tree storage plugin in Kubernetes. Hence, you only need to install the storage class.

This tutorial demonstrates how to use KubeKey to set up a Kube AI Hub cluster and configure GlusterFS to provide storage services.

Prerequisites

You have set up your GlusterFS cluster and configured Heketi. For more information, see Set up a GlusterFS Server.

Step 1: Configure the Client Machine

You need to install the GlusterFS client package on all your client machines.

1. Install software-properties-common.

   apt-get install software-properties-common
   

2. Add the community GlusterFS PPA.

   add-apt-repository ppa:gluster/glusterfs-7
   

3. Make sure you are using the latest package.

   apt-get update
   

4. Install the GlusterFS client.

   apt-get install glusterfs-server -y
   

5. Verify your GlusterFS version.

   glusterfs -V
   

Step 2: Create a Configuration File for GlusterFS

The separate configuration file contains all parameters of GlusterFS storage which will be used by KubeKey during installation.

1. Go to one of the nodes (taskbox) where you want to download KubeKey later and run the following command to create a configuration file.

   vi glusterfs-sc.yaml
   

   An example configuration file (include a Heketi Secret):

   apiVersion: v1
   kind: Secret
   metadata:
     name: heketi-secret
     namespace: kube-system
   type: kubernetes.io/glusterfs
   data:
     key: "MTIzNDU2"    # Replace it with your own key. Base64 coding.
   ---
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     annotations:
       storageclass.beta.kubernetes.io/is-default-class: "true"
       storageclass.kubesphere.io/supported-access-modes: '["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]'
     name: glusterfs
   parameters:
     clusterid: "21240a91145aee4d801661689383dcd1"    # Replace it with your own GlusterFS cluster ID.
     gidMax: "50000"
     gidMin: "40000"
     restauthenabled: "true"
     resturl: "http://192.168.0.2:8080"    # The Gluster REST service/Heketi service url which provision gluster volumes on demand. Replace it with your own.
     restuser: admin
     secretName: heketi-secret
     secretNamespace: kube-system
     volumetype: "replicate:3"    # Replace it with your own volume type.
   provisioner: kubernetes.io/glusterfs
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   allowVolumeExpansion: true
   

   Note

   • Use the field storageclass.beta.kubernetes.io/is-default-class to set glusterfs as your default storage class. If it is false, KubeKey will install OpenEBS as the default storage class.
   • For more information about parameters in the storage class manifest, see the Kubernetes documentation.

2. Save the file.

Step 3: Download KubeKey

Follow the steps below to download KubeKey on the taskbox.

curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -

export KKZONE=cn

curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -

Make kk executable:

chmod +x kk

Step 4: Create a Cluster

1. Specify a Kubernetes version and a Kube AI Hub version that you want to install. For example:

   ./kk create config --with-kubernetes v1.22.12 --with-kubesphere v3.4.1
   

   Note

   • Recommended Kubernetes versions for Kube AI Hub 3.4: v1.20.x, v1.21.x, v1.22.x, v1.23.x, * v1.24.x, * v1.25.x, and * v1.26.x. For Kubernetes versions with an asterisk, some features of edge nodes may be unavailable due to incompatability. Therefore, if you want to use edge nodes, you are advised to install Kubernetes v1.23.x. If you do not specify a Kubernetes version, KubeKey will install Kubernetes v1.23.10 by default. For more information about supported Kubernetes versions, see Support Matrix.

   • If you do not add the flag --with-kubesphere in the command in this step, Kube AI Hub will not be deployed unless you install it using the addons field in the configuration file or add this flag again when you use ./kk create cluster later.

   • If you add the flag --with-kubesphere without specifying a Kube AI Hub version, the latest version of Kube AI Hub will be installed.

2. A default file config-sample.yaml will be created if you do not customize the name. Edit the file.

   vi config-sample.yaml
   

   ...
   metadata:
     name: sample
   spec:
     hosts:
     - {name: client1, address: 192.168.0.5, internalAddress: 192.168.0.5, user: ubuntu, password: Testing123}
     - {name: client2, address: 192.168.0.6, internalAddress: 192.168.0.6, user: ubuntu, password: Testing123}
     - {name: client3, address: 192.168.0.7, internalAddress: 192.168.0.7, user: ubuntu, password: Testing123}
     roleGroups:
       etcd:
       - client1
       control-plane:
       - client1
       worker:
       - client2
       - client3
     controlPlaneEndpoint:
       domain: lb.kubesphere.local
       address: ""
       port: 6443
     kubernetes:
       version: v1.22.12
       imageRepo: kubesphere
       clusterName: cluster.local
     network:
       plugin: calico
       kubePodsCIDR: 10.233.64.0/18
       kubeServiceCIDR: 10.233.0.0/18
     registry:
       registryMirrors: []
       insecureRegistries: []
     addons:
     - name: glusterfs
       namespace: kube-system
       sources:
         yaml:
           path:
           - /root/glusterfs-sc.yaml
   ...
   

3. Pay special attention to the field of addons, under which you must provide the information of the storage class to be created as well as the Heketi Secret. For more information about each parameter in this file, see Multi-node Installation.

4. Save the file and execute the following command to install Kubernetes and Kube AI Hub:

   ./kk create cluster -f config-sample.yaml
   

5. When the installation finishes, you can inspect installation logs with the following command:

   kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
   

   Expected output:

   #####################################################
   ###              Welcome to Kube AI Hub!           ###
   #####################################################
   
   Console: http://192.168.0.4:30880
   Account: admin
   Password: P@88w0rd
   
   NOTES：
     1. After you log into the console, please check the
        monitoring status of service components in
        "Cluster Management". If any service is not
        ready, please wait patiently until all components 
        are up and running.
     2. Please change the default password after login.
   
   #####################################################
   https://kubesphere.io             20xx-xx-xx xx:xx:xx
   #####################################################
   

Step 5: Verify Installation

You can verify that GlusterFS has been successfully installed either from the command line or from the Kube AI Hub web console.

Command line

Run the following command to check your storage class.

kubectl get sc

Expected output:

NAME                  PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
glusterfs (default)   kubernetes.io/glusterfs   Delete          Immediate           true                   104m

Kube AI Hub console

1. Log in to the web console with the default account and password (admin/P@88w0rd) at <NodeIP>:30880. Click Platform in the upper-left corner and select Cluster Management.

2. Go to Volumes under Storage, and you can see PVCs in use.

   Note

   For more information about how to create volumes on the Kube AI Hub console, see Volumes.

3. On the Storage Classes page, you can see the storage class available in your cluster.