Last updated:

Air-gapped Installation

KubeKey is an open-source, lightweight tool for deploying Kubernetes clusters. It allows you to install Kubernetes/K3s only, both Kubernetes/K3s and Kube AI Hub, and other cloud-native plugins in a flexible, fast, and convenient way. Additionally, it is an effective tool for scaling and upgrading clusters.

In KubeKey v2.1.0, we bring in concepts of manifest and artifact, which provides a solution for air-gapped installation of Kubernetes clusters. A manifest file describes information of the current Kubernetes cluster and defines content in an artifact. Previously, users had to prepare deployment tools, image (.tar) file, and other binaries as the Kubernetes version and image to deploy are different. Now, with KubeKey, air-gapped installation can never be so easy. You simply use a manifest file to define what you need for your cluster in air-gapped environments, and then export the artifact file to quickly and easily deploy image registries and Kubernetes cluster.

Prerequisites

Host IP	Host Name	Usage
192.168.0.2	node1	Online host for packaging the source cluster
192.168.0.3	node2	Control plane node of the air-gapped environment
192.168.0.4	node3	Image registry node of the air-gapped environment

Preparations

Run the following commands to download KubeKey.
Download KubeKey from its GitHub Release Page or use the following command directly.
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -
Run the following command first to make sure you download KubeKey from the correct zone.
export KKZONE=cn
Run the following command to download KubeKey:
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -

On the online host, run the following command and copy content in the manifest-example.

vim manifest.yaml

---
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
    - amd64
  operatingSystems:
    - arch: amd64
      type: linux
      id: centos
      version: "7"
      repository:
        iso:
          localPath:
          url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/centos7-rpms-amd64.iso
    - arch: amd64
      type: linux
      id: ubuntu
      version: "20.04"
      repository:
        iso:
          localPath:
          url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/ubuntu-20.04-debs-amd64.iso
  kubernetesDistributions:
    - type: kubernetes
      version: v1.23.15
  components:
    helm:
      version: v3.9.0
    cni:
      version: v1.2.0
    etcd:
      version: v3.4.13
    calicoctl:
      version: v3.23.2
    ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
    ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
    containerRuntimes:
      - type: docker
        version: 20.10.8
      - type: containerd
        version: 1.6.4
    crictl:
      version: v1.24.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.5.3
    docker-compose:
      version: v2.2.2
  images:
    - docker.io/kubesphere/kube-apiserver:v1.23.15
    - docker.io/kubesphere/kube-controller-manager:v1.23.15
    - docker.io/kubesphere/kube-proxy:v1.23.15
    - docker.io/kubesphere/kube-scheduler:v1.23.15
    - docker.io/kubesphere/pause:3.6
    - docker.io/coredns/coredns:1.8.6
    - docker.io/calico/cni:v3.23.2
    - docker.io/calico/kube-controllers:v3.23.2
    - docker.io/calico/node:v3.23.2
    - docker.io/calico/pod2daemon-flexvol:v3.23.2
    - docker.io/calico/typha:v3.23.2
    - docker.io/kubesphere/flannel:v0.12.0
    - docker.io/openebs/provisioner-localpv:3.3.0
    - docker.io/openebs/linux-utils:3.3.0
    - docker.io/library/haproxy:2.3
    - docker.io/kubesphere/nfs-subdir-external-provisioner:v4.0.2
    - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
    - docker.io/kubesphere/ks-installer:v3.4.1
    - docker.io/kubesphere/ks-apiserver:v3.4.1
    - docker.io/kubesphere/ks-console:v3.4.1
    - docker.io/kubesphere/ks-controller-manager:v3.4.1
    - docker.io/kubesphere/kubectl:v1.22.0
    - docker.io/kubesphere/kubectl:v1.21.0
    - docker.io/kubesphere/kubectl:v1.20.0
    - docker.io/kubesphere/kubefed:v0.8.1
    - docker.io/kubesphere/tower:v0.2.1
    - docker.io/minio/minio:RELEASE.2019-08-07T01-59-21Z
    - docker.io/minio/mc:RELEASE.2019-08-07T23-14-43Z
    - docker.io/csiplugin/snapshot-controller:v4.0.0
    - docker.io/kubesphere/nginx-ingress-controller:v1.1.0
    - docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.4
    - docker.io/kubesphere/metrics-server:v0.4.2
    - docker.io/library/redis:5.0.14-alpine
    - docker.io/library/haproxy:2.0.25-alpine
    - docker.io/library/alpine:3.14
    - docker.io/osixia/openldap:1.3.0
    - docker.io/kubesphere/netshoot:v1.0
    - docker.io/kubeedge/cloudcore:v1.13.0
    - docker.io/kubeedge/iptables-manager:v1.13.0
    - docker.io/kubesphere/edgeservice:v0.3.0
    - docker.io/openpolicyagent/gatekeeper:v3.5.2
    - docker.io/kubesphere/openpitrix-jobs:v3.3.2
    - docker.io/kubesphere/devops-apiserver:ks-v3.4.1
    - docker.io/kubesphere/devops-controller:ks-v3.4.1
    - docker.io/kubesphere/devops-tools:ks-v3.4.1
    - docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1
    - docker.io/jenkins/inbound-agent:4.10-2
    - docker.io/kubesphere/builder-base:v3.2.2
    - docker.io/kubesphere/builder-nodejs:v3.2.0
    - docker.io/kubesphere/builder-maven:v3.2.0
    - docker.io/kubesphere/builder-maven:v3.2.1-jdk11
    - docker.io/kubesphere/builder-python:v3.2.0
    - docker.io/kubesphere/builder-go:v3.2.0
    - docker.io/kubesphere/builder-go:v3.2.2-1.16
    - docker.io/kubesphere/builder-go:v3.2.2-1.17
    - docker.io/kubesphere/builder-go:v3.2.2-1.18
    - docker.io/kubesphere/builder-base:v3.2.2-podman
    - docker.io/kubesphere/builder-nodejs:v3.2.0-podman
    - docker.io/kubesphere/builder-maven:v3.2.0-podman
    - docker.io/kubesphere/builder-maven:v3.2.1-jdk11-podman
    - docker.io/kubesphere/builder-python:v3.2.0-podman
    - docker.io/kubesphere/builder-go:v3.2.0-podman
    - docker.io/kubesphere/builder-go:v3.2.2-1.16-podman
    - docker.io/kubesphere/builder-go:v3.2.2-1.17-podman
    - docker.io/kubesphere/builder-go:v3.2.2-1.18-podman
    - docker.io/kubesphere/s2ioperator:v3.2.1
    - docker.io/kubesphere/s2irun:v3.2.0
    - docker.io/kubesphere/s2i-binary:v3.2.0
    - docker.io/kubesphere/tomcat85-java11-centos7:v3.2.0
    - docker.io/kubesphere/tomcat85-java11-runtime:v3.2.0
    - docker.io/kubesphere/tomcat85-java8-centos7:v3.2.0
    - docker.io/kubesphere/tomcat85-java8-runtime:v3.2.0
    - docker.io/kubesphere/java-11-centos7:v3.2.0
    - docker.io/kubesphere/java-8-centos7:v3.2.0
    - docker.io/kubesphere/java-8-runtime:v3.2.0
    - docker.io/kubesphere/java-11-runtime:v3.2.0
    - docker.io/kubesphere/nodejs-8-centos7:v3.2.0
    - docker.io/kubesphere/nodejs-6-centos7:v3.2.0
    - docker.io/kubesphere/nodejs-4-centos7:v3.2.0
    - docker.io/kubesphere/python-36-centos7:v3.2.0
    - docker.io/kubesphere/python-35-centos7:v3.2.0
    - docker.io/kubesphere/python-34-centos7:v3.2.0
    - docker.io/kubesphere/python-27-centos7:v3.2.0
    - quay.io/argoproj/argocd:v2.3.3
    - quay.io/argoproj/argocd-applicationset:v0.4.1
    - ghcr.io/dexidp/dex:v2.30.2
    - docker.io/library/redis:6.2.6-alpine
    - docker.io/jimmidyson/configmap-reload:v0.7.1
    - docker.io/prom/prometheus:v2.39.1
    - docker.io/kubesphere/prometheus-config-reloader:v0.55.1
    - docker.io/kubesphere/prometheus-operator:v0.55.1
    - docker.io/kube-ai-hub/kube-rbac-proxy:v0.14.0
    - docker.io/kubesphere/kube-state-metrics:v2.6.0
    - docker.io/prom/node-exporter:v1.3.1
    - docker.io/prom/alertmanager:v0.23.0
    - docker.io/thanosio/thanos:v0.31.0
    - docker.io/grafana/grafana:8.3.3
    - docker.io/kube-ai-hub/kube-rbac-proxy:v0.14.0
    - docker.io/kubesphere/notification-manager-operator:v2.3.0
    - docker.io/kubesphere/notification-manager:v2.3.0
    - docker.io/kubesphere/notification-tenant-sidecar:v3.2.0
    - docker.io/kubesphere/elasticsearch-curator:v5.7.6
    - docker.io/kubesphere/elasticsearch-oss:6.8.22
    - docker.io/opensearchproject/opensearch:2.6.0
    - docker.io/opensearchproject/opensearch-dashboards:2.6.0
    - docker.io/kubesphere/opensearch-curator:v0.0.5
    - docker.io/kubesphere/fluentbit-operator:v0.14.0
    - docker.io/library/docker:19.03
    - docker.io/kubesphere/fluent-bit:v1.9.4
    - docker.io/kubesphere/log-sidecar-injector:v1.2.0
    - docker.io/elastic/filebeat:6.7.0
    - docker.io/kubesphere/kube-events-operator:v0.6.0
    - docker.io/kubesphere/kube-events-exporter:v0.6.0
    - docker.io/kubesphere/kube-events-ruler:v0.6.0
    - docker.io/kubesphere/kube-auditing-operator:v0.2.0
    - docker.io/kubesphere/kube-auditing-webhook:v0.2.0
    - docker.io/istio/pilot:1.14.6
    - docker.io/istio/proxyv2:1.14.6
    - docker.io/jaegertracing/jaeger-operator:1.29
    - docker.io/jaegertracing/jaeger-agent:1.29
    - docker.io/jaegertracing/jaeger-collector:1.29
    - docker.io/jaegertracing/jaeger-query:1.29
    - docker.io/jaegertracing/jaeger-es-index-cleaner:1.29
    - docker.io/kubesphere/kiali-operator:v1.50.1
    - docker.io/kubesphere/kiali:v1.50
    - docker.io/library/busybox:1.31.1
    - docker.io/library/nginx:1.14-alpine
    - docker.io/joosthofman/wget:1.0
    - docker.io/nginxdemos/hello:plain-text
    - docker.io/library/wordpress:4.8-apache
    - docker.io/mirrorgooglecontainers/hpa-example:latest
    - docker.io/fluent/fluentd:v1.4.2-2.0
    - docker.io/library/perl:latest
    - docker.io/kubesphere/examples-bookinfo-productpage-v1:1.16.2
    - docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2
    - docker.io/kubesphere/examples-bookinfo-reviews-v2:1.16.2
    - docker.io/kubesphere/examples-bookinfo-details-v1:1.16.2
    - docker.io/kubesphere/examples-bookinfo-ratings-v1:1.16.3
    - docker.io/weaveworks/scope:1.13.0

Note

If the artifact file to export contains ISO dependencies, such as conntarck and chrony, set the IP address for downloading the ISO dependencies in .repostiory.iso.url of operationSystem. Alternatively, you can download the ISO package in advance and fill in the local path in localPath and delete the url configuration item.
You need to enable harbor and docker-compose configuration items, which will be used when you use KubeKey to build a Harbor registry for pushing images.
By default, the list of images in the created manifest is obtained from docker.io.
You can customize the manifest-sample.yaml file to export the desired artifact file.
You can download the ISO files at https://github.com/kubesphere/kubekey/releases/tag/v3.0.7.

If you already deployed a cluster, you can run the following command in the cluster to create a manifest file and configure the file according to the sample in Step 2.
```
./kk create manifest
```
Export the artifact.
Run the following command directly:
./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz
Run the following commands:
export KKZONE=cn ./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz
Note
An artifact is a .tgz package containing the image package (.tar) and associated binaries exported from the specified manifest file. You can specify an artifact in the KubeKey commands for initializing the image registry, creating clusters, adding nodes, and upgrading clusters, and then KubeKey will automatically unpack the artifact and use the unpacked file when running the command.
- Make sure the network connection is working.
- KubeKey will resolve image names in the image list. If the image registry requires authentication, you can configure it in .registry.auths in the manifest file.

Install Clusters in the Air-gapped Environment

Copy the downloaded KubeKey and artifact to nodes in the air-gapped environment using a USB device.

Run the following command to create a configuration file for the air-gapped cluster:

./kk create config --with-kubesphere v3.4.1 --with-kubernetes v1.23.15 -f config-sample.yaml

Run the following command to modify the configuration file:

vim config-sample.yaml

Note

Modify the node information according to the actual configuration of the air-gapped environment.
You must specify the node where the registry to deploy (for KubeKey deployment of self-built Harbor registries).
In registry, the value of type must be specified as that of harbor. Otherwise, the docker registry is installed by default.

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
    - {
        name: master,
        address: 192.168.0.3,
        internalAddress: 192.168.0.3,
        user: root,
        password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>",
      }
    - {
        name: node1,
        address: 192.168.0.4,
        internalAddress: 192.168.0.4,
        user: root,
        password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>",
      }

  roleGroups:
    etcd:
      - master
    control-plane:
      - master
    worker:
      - node1
    # If you want to use KubeKey to automatically deploy the image registry, set this value. You are advised to separately deploy the registry and the cluster.
    registry:
      - node1
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.23.15
    clusterName: cluster.local
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    # To use KubeKey to deploy Harbor, set the value of this parameter to harbor. If you do not set this parameter and still use KubeKey to create an container image registry, the docker registry is used by default.
    type: harbor
    # If Harbor or other registries deployed by using KubeKey requires login, you can set the auths parameter of the registry. However, if you create a docker registry using KubeKey, you do not need to set the auths parameter.
    # Note: If you use KubeKey to deploy Harbor, do not set this parameter until Harbor is started.
    #auths:
    #  "dockerhub.kubekey.local":
    #    username: admin
    #    password: Harbor12345
    # Set the private registry to use during cluster deployment.
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []

Run the following command to install an image registry:
```
./kk init registry -f config-sample.yaml -a kubesphere.tar.gz
```
Note
The parameters in the command are explained as follows:
- config-sample.yaml: Specifies the configuration file of the cluster in the air-gapped environment.
- kubesphere.tar.gz: Specifies the image package of the source cluster.

Create a Harbor project.

Note

As Harbor adopts the Role-based Access Control (RBAC) mechanism, which means that only specified users can perform certain operations. Therefore, you must create a project before pushing images to Harbor. Harbor supports two types of projects:

Public: All users can pull images from the project.
Private: Only project members can pull images from the project.

The username and password for logging in to Harbor is admin and Harbor12345 by default. The installation file of Harbor is located in /opt/harbor, where you can perform O&M of Harbor.

Method 1: Run the following commands to create a Harbor project.

a. Run the following command to download the specified script to initialize the Harbor registry:

curl -O https://raw.githubusercontent.com/kubesphere/ks-installer/master/scripts/create_project_harbor.sh

b. Run the following command to modify the script configuration file:

vim create_project_harbor.sh

#!/usr/bin/env bash

# Copyright 2018 The Kube AI Hub Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

url="https://dockerhub.kubekey.local"  #Change the value of url to https://dockerhub.kubekey.local.
user="admin"
passwd="Harbor12345"

harbor_projects=(library
    kubesphereio
    kubesphere
    argoproj
    calico
    coredns
    openebs
    csiplugin
    minio
    mirrorgooglecontainers
    osixia
    prom
    thanosio
    jimmidyson
    grafana
    elastic
    istio
    jaegertracing
    jenkins
    weaveworks
    openpitrix
    joosthofman
    nginxdemos
    fluent
    kubeedge
    openpolicyagent
)

for project in "${harbor_projects[@]}"; do
    echo "creating $project"
    curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k #Add -k at the end of the curl command.
done

Note

Change the value of url to https://dockerhub.kubekey.local.
The project name of the registry must be the same as that of the image list.
Add -k at the end of the curl command.

c. Run the following commands to create a Harbor project:

chmod +x create_project_harbor.sh

./create_project_harbor.sh

Method 2: Log in to Harbor and create a project. Set the project to Public, so that any user can pull images from this project. For more information, please refer to Create Projects.

harbor-login

Run the following command again to modify the cluster configuration file：

vim config-sample.yaml

Note

In auths, add dockerhub.kubekey.local and the username and password.
In privateRegistry, add dockerhub.kubekey.local.

  ...
  registry:
    type: harbor
    auths:
      "dockerhub.kubekey.local":
        username: admin
        password: Harbor12345
    privateRegistry: "dockerhub.kubekey.local"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
  addons: []

Note

In auths, enter dockerhub.kubekey.local, username (admin) and password (Harbor12345).
In privateRegistry, enter dockerhub.kubekey.local.
In namespaceOverride, enter kubesphereio.

Run the following command to install a Kube AI Hub cluster:
```
./kk create cluster -f config-sample1.yaml -a kubesphere.tar.gz --with-packages
```
The parameters are explained as follows：
- config-sample.yaml: Specifies the configuration file for the cluster in the air-gapped environment.
- kubesphere.tar.gz: Specifies the tarball image from which the source cluster is packaged.
- --with-packages: This parameter is required if you want to install the ISO dependencies.

Run the following command to view the cluster status:

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

After the installation is completed, the following information is displayed:

**************************************************
#####################################################
###              Welcome to Kube AI Hub!           ###
#####################################################

Console: http://192.168.0.3:30880
Account: admin
Password: P@88w0rd

NOTES：
1. After you log into the console, please check the
monitoring status of service components in
the "Cluster Management". If any service is not
ready, please wait patiently until all components
are up and running.
1. Please change the default password after login.

#####################################################
https://kubesphere.io             2022-02-28 23:30:06
#####################################################

Access Kube AI Hub's web console at http://{IP}:30880 using the default account and password admin/P@88w0rd.

Note

To access the console, make sure that port 30880 is enabled in your security group.

Previous : Multi-node Installation Next : Port Requirements

What’s on this Page