Install Huawei Ascend Driver

This guide describes how to install Huawei Ascend NPU drivers, firmware, and Ascend Docker Runtime on Linux NPU nodes so that Kube AI Hub can schedule Ascend NPU workloads. Complete these steps on all NPU worker nodes before KubeKey creates the cluster.

Prerequisites

• Node has Huawei Ascend NPUs (for example 910C, 910B, 310P / 310P48; verify against your hardware)
• root or sudo privileges
• Time synchronization is configured
• Driver, firmware, and runtime packages matching your NPU model, OS, and architecture are prepared

Installation Order

Installation Media

Replace {model}, {version}, and {arch} with actual values (e.g., 910b, x86_64).

Download links:

• Ascend Community Drivers and Firmware
• Ascend Docker Runtime (Gitee)

Create the Runtime User

Ascend driver installation requires the HwHiAiUser user:

sudo groupadd HwHiAiUser 2>/dev/null || true
sudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash 2>/dev/null || true

Install Dependencies

RHEL / CentOS / Rocky and similar:

sudo yum install -y dkms gcc make kernel-headers-$(uname -r) kernel-devel-$(uname -r)

Ubuntu / Debian:

sudo apt-get update
sudo apt-get install -y dkms gcc make linux-headers-$(uname -r)

Install the NPU Driver

Make the package executable and verify integrity:

chmod +x Ascend-hdk-{model}-npu-driver_{version}_linux-{arch}.run
./Ascend-hdk-{model}-npu-driver_{version}_linux-{arch}.run --check

OK output indicates the package is intact. Install:

sudo ./Ascend-hdk-{model}-npu-driver_{version}_linux-{arch}.run --full

Install NPU Firmware

chmod +x Ascend-hdk-{model}-npu-firmware_{version}.run
./Ascend-hdk-{model}-npu-firmware_{version}.run --check
sudo ./Ascend-hdk-{model}-npu-firmware_{version}.run --full

Confirm firmware version matches the package:

/usr/local/Ascend/driver/tools/upgrade-tool --device_index -1 --component -1 --version

Verify the Driver

npu-smi info

Successful NPU information output indicates the driver is installed correctly.

Hard-slice prerequisites: AVI and templates

Whole-card scheduling does not depend on vNPU template names. For memory-based hard slicing, enable Ascend Virtualization Instance (AVI) in container mode on the host, and confirm the templates this card actually supports.

# 0 = docker / container mode
npu-smi set -t vnpu-mode -d 0
npu-smi info -t vnpu-mode
# expect: vnpu-mode : docker

npu-smi info -m
npu-smi info -t template-info -i <NPU_ID> -c <CHIP_ID>

template-info requires -i / -c (IDs from npu-smi info -m). Omitting them can print "not supported" even when AVI is on.

Atlas A3 (910C) training and inference cards use different template names. The cluster default is training (vir06_1c_16g / vir12_3c_32g). If the query returns vir05_1c_16g / vir10_3c_32g, change the HAMi ConfigMap after install. See Ascend NPU Usage.

Install Ascend Docker Runtime

You must install Ascend Docker Runtime. The cluster injects runtimeClassName: ascend on NPU Pods by default so this runtime can mount device nodes and driver libraries. Without it, containers cannot use the NPU and npu-smi fails.

Verify and install the runtime package:

chmod +x Ascend-docker-runtime_{version}_linux-{arch}.run
./Ascend-docker-runtime_{version}_linux-{arch}.run --check
sudo ./Ascend-docker-runtime_{version}_linux-{arch}.run --install

Configure containerd

Edit /etc/containerd/config.toml and point the runtime to Ascend Docker Runtime. In the [plugins."io.containerd.runtime.v1.linux"] section:

[plugins."io.containerd.runtime.v1.linux"]
  no_shim = false
  runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
  runtime_root = ""
  shim = "containerd-shim"
  shim_debug = false

Restart containerd to apply the configuration:

sudo systemctl daemon-reload
sudo systemctl restart containerd

KubeKey / containerd 1.7 CRI also needs a named runtime ascend, or the RuntimeClass handler will not resolve. Add this under [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] (keep the existing runc runtime):

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend.options]
    BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
    SystemdCgroup = true

On NPU-only nodes you may set default_runtime_name = "ascend". On mixed NVIDIA + Ascend nodes keep runc as the default and use RuntimeClass only on NPU Pods.

Restart containerd again after this change.

Create the RuntimeClass

The cluster must have a RuntimeClass named ascend. Otherwise Pods fail with RuntimeClass "ascend" not found. The HAMi ascend-device-plugin subchart creates it by default. If it is missing:

kubectl apply -f - <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: ascend
handler: ascend
EOF

kubectl get runtimeclass ascend

handler must match the containerd runtime name after runtimes. (ascend).

Verification Checklist

Troubleshooting

--check verification fails

• Re-download the package and confirm it was not corrupted during transfer
• Confirm the package matches your hardware model and OS architecture

npu-smi fails or produces no output

• Confirm correct install order (first-time: driver → firmware)
• Confirm the HwHiAiUser user exists
• Verify kernel headers match the running kernel: uname -r

Containers cannot use the NPU

• Confirm the runtime path in /etc/containerd/config.toml is correct
• Confirm containerd was restarted
• Consult Ascend official documentation for CANN/driver and container image compatibility
• After the cluster is up, request whole cards or hard-template slices as described in Ascend NPU Usage
• Hard-slice Pods fail immediately with create-vnpu in the log: compare template-info with the ConfigMap and keep both on training templates or both on inference templates