Bring Your Own Prometheus

Kube AI Hub comes with several pre-installed customized monitoring components, including Prometheus Operator, Prometheus, Alertmanager, Grafana (Optional), various ServiceMonitors, node-exporter, and kube-state-metrics. These components might already exist before you install Kube AI Hub. It is possible to use your own Prometheus stack setup in Kube AI Hub v3.4.

Bring Your Own Prometheus

Step 1. Uninstall the customized Prometheus stack of Kube AI Hub

1. Execute the following commands to uninstall the stack:

   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/alertmanager/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/devops/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/etcd/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/grafana/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/kube-state-metrics/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/node-exporter/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/upgrade/ 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/prometheus-rules-v1.16\+.yaml 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/prometheus-rules.yaml 2>/dev/null
   kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/prometheus 2>/dev/null
   # Uncomment this line if you don't have Prometheus managed by Prometheus Operator in other namespaces.
   # kubectl -n kubesphere-system exec $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}') -- kubectl delete -f /kubesphere/kubesphere/prometheus/init/ 2>/dev/null
   

2. Delete the PVC that Prometheus uses.

   kubectl -n kubesphere-monitoring-system delete pvc `kubectl -n kubesphere-monitoring-system get pvc | grep -v VOLUME | awk '{print $1}' |  tr '\n' ' '`
   

Step 2. Install your own Prometheus stack

The Prometheus stack can be installed in many ways. The following steps show how to install it into the namespace monitoring using Prometheus stack manifests in ks-installer (generated from a Kube AI Hub custom version of kube-prometheus).

1. Obtain ks-installer that Kube AI Hub v3.4 uses.

   cd ~ && git clone -b release-3.4 https://github.com/kubesphere/ks-installer.git && cd ks-installer/roles/ks-monitor/files/prometheus
   

2. Create kustomization.yaml:

   # create
   cat <<EOF > kustomization.yaml
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: monitoring
   resources:
   EOF
   
   # append yaml paths
   find . -mindepth 2 -name "*.yaml" -type f -print | sed 's/^/- /' >> kustomization.yaml
   

3. Remove unnecessary components. For example, if Grafana is not enabled in Kube AI Hub, you can run the following command to delete the Grafana section in kustomization.yaml.

   sed -i '/grafana\//d' kustomization.yaml
   

4. Install the stack.

   kubectl apply -k .
   

Step 3. Install Kube AI Hub customized stuff to your Prometheus stack

1. Obtain ks-installer that Kube AI Hub v3.4 uses.

   cd ~ && git clone -b release-3.4 https://github.com/kubesphere/ks-installer.git && cd ks-installer/roles/ks-monitor/files/prometheus
   

2. Create kustomization.yaml, fill the following content.

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: <your own namespace>
   resources:
     - ./alertmanager/alertmanager-secret.yaml
     - ./etcd/prometheus-rulesEtcd.yaml
     - ./kube-state-metrics/kube-state-metrics-serviceMonitor.yaml
     - ./kubernetes/kubernetes-prometheusRule.yaml
     - ./kubernetes/kubernetes-serviceKubeControllerManager.yaml
     - ./kubernetes/kubernetes-serviceKubeScheduler.yaml
     - ./kubernetes/kubernetes-serviceMonitorApiserver.yaml
     - ./kubernetes/kubernetes-serviceMonitorCoreDNS.yaml
     - ./kubernetes/kubernetes-serviceMonitorKubeControllerManager.yaml
     - ./kubernetes/kubernetes-serviceMonitorKubeScheduler.yaml
     - ./kubernetes/kubernetes-serviceMonitorKubelet.yaml
     - ./node-exporter/node-exporter-serviceMonitor.yaml
     - ./prometheus/prometheus-clusterRole.yaml
   

   Note

   • Set the value of namespace to your own namespace in which the Prometheus stack is deployed. For example, it is monitoring if you install Prometheus in the monitoring namespace in Step 2.
   • If you have enabled the alerting component for Kube AI Hub, supplement yaml paths of thanos-ruler into kustomization.yaml.

3. Install the required components of Kube AI Hub.

   kubectl apply -k .
   

4. Find the Prometheus CR which is usually k8s in your own namespace.

   kubectl -n <your own namespace> get prometheus
   

5. Set the Prometheus rule evaluation interval to 1m to be consistent with the Kube AI Hub v3.4 customized ServiceMonitor. The Rule evaluation interval should be greater than or equal to the scrape interval.

   kubectl -n <your own namespace> patch prometheus k8s --patch '{
     "spec": {
       "evaluationInterval": "1m"
     }
   }' --type=merge
   

Step 4. Change Kube AI Hub's monitoring endpoint

Now that your own Prometheus stack is up and running, you can change Kube AI Hub's monitoring endpoint to use your own Prometheus.

1. Run the following command to edit kubesphere-config.

   kubectl edit cm -n kubesphere-system kubesphere-config
   

2. Navigate to the monitoring endpoint section, as shown in the following:

       monitoring:
         endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
   

3. Change monitoring endpoint to your own Prometheus:

       monitoring:
         endpoint: http://prometheus-operated.monitoring.svc:9090
   

4. If you have enabled the alerting component of Kube AI Hub, navigate to prometheusEndpoint and thanosRulerEndpoint of alerting, and change the values according to the following sample. Kube AI Hub APIServer will restart automatically to make your configurations take effect.

   ...
      alerting:
        ...
        prometheusEndpoint: http://prometheus-operated.monitoring.svc:9090
        thanosRulerEndpoint: http://thanos-ruler-operated.monitoring.svc:10902
        ...
   ...