Ascend NPU Resource Usage

Kube AI Hub schedules Ascend NPUs through HAMi. The production model is Ascend Runtime + hard-template slicing. Install the Ascend driver and Ascend Docker Runtime on NPU nodes first. See GPU Virtualization Mode for the slicing model and NPU monitoring for metrics.

How to request an NPU

Workloads use hami-scheduler. If runtimeClassName is omitted, the admission webhook injects ascend so the container can mount /dev/davinci* and the driver libraries.

Supported models and resource names

The console GPU Type list and YAML use the same resource names. Count and memory must belong to the same SKU.

Hard templates

The scheduler rounds *-memory up to the smallest template that is at least as large as the request. Inside the container, npu-smi info shows the VF / template name.

The cluster default is Atlas A3 training templates. 16384 maps to vir06_1c_16g; 20000–32768 maps to vir12_3c_32g. On an A3 inference card you must switch to the inference templates in the next section, or Runtime create-vnpu fails.

Atlas A3 (910C) training vs inference

Training and inference 910C cards share the resource name huawei.com/Ascend910C and about 64 GiB of HBM, but the driver exposes different vNPU template names. HAMi picks one name from the ConfigMap templates list by memory only. Do not enable both sets under the same Ascend910C entry — the 16G / 32G names collide.

Trust the driver on the node, not the marketing name:

npu-smi info -m
# Use NPU ID and Chip ID from that table:
npu-smi info -t vnpu-mode
npu-smi info -t template-info -i <NPU_ID> -c <CHIP_ID>

• Before hard slicing, set AVI to container mode on the host: npu-smi set -t vnpu-mode -d 0. Query should show vnpu-mode : docker.
• Whole-card requests (no *-memory) do not need a matching template name. Training and inference cards both work.
• Memory must be an integer MiB value (16384 / 32768). Do not use 16Gi or 24k.

Switching an inference-only cluster

The Chart keeps training templates by default. After you install on inference nodes, edit ConfigMap hami-scheduler-device in kube-system, key device-config.yaml, and replace the templates list under commonWord: Ascend910C with:

        templates:
          - name: vir05_1c_16g
            memory: 16384
            aiCore: 5
            aiCPU: 1
          - name: vir10_3c_32g
            memory: 32768
            aiCore: 10
            aiCPU: 3

The Helm source HAMi/charts/hami/templates/scheduler/device-configmap.yaml already has those inference templates in comments: uncomment them and comment out the training pair. Then restart the scheduler (and preferably the device plugin):

kubectl -n kube-system rollout restart deploy/hami-scheduler
kubectl -n kube-system rollout restart ds/hami-ascend-device-plugin

Persist the change in the Chart or overlay, then helm upgrade. Editing only the live ConfigMap is overwritten by the next Helm upgrade.

Hard-slicing both training and inference cards in one cluster is not supported on a single Ascend910C template list. Split by card type, or schedule inference cards as whole cards.

910C and 310P rules

910C

• A whole-card request of 1 is adjusted to 2 (one physical module = a pair of dies).
• A slice with *-memory below whole-card memory keeps count 1.
• Odd whole-card counts (3, 5, 7, …) are rejected. Multi-NPU whole-card allocations use pairs 0+1, 2+3, and so on.

310P

• The device plugin picks the resource name from each die's DCMI memory: below 32768 MiB → Ascend310P; 32768 MiB or more → Ascend310P48.
• A pod that requests huawei.com/Ascend310P will not schedule on a 48G-die node. Use Ascend310P48 instead.

Examples

Whole card (910C; the webhook changes 1 to 2 and fills memory):

apiVersion: v1
kind: Pod
metadata:
  name: ascend910c-whole
spec:
  schedulerName: hami-scheduler
  containers:
    - name: inference
      image: your-registry/vllm-ascend:latest
      resources:
        limits:
          huawei.com/Ascend910C: "1"

32G hard slice (910C training default → vir12_3c_32g; inference templates map the same MiB to vir10_3c_32g):

apiVersion: v1
kind: Pod
metadata:
  name: ascend910c-slice-32g
spec:
  schedulerName: hami-scheduler
  containers:
    - name: inference
      image: your-registry/vllm-ascend:latest
      resources:
        limits:
          huawei.com/Ascend910C: "1"
          huawei.com/Ascend910C-memory: "32768"

310P48 hard slice (8000 rounds up to vir02 / 12288 MiB):

apiVersion: v1
kind: Pod
metadata:
  name: ascend310p48-slice
spec:
  schedulerName: hami-scheduler
  containers:
    - name: inference
      image: your-registry/vllm-ascend:latest
      resources:
        limits:
          huawei.com/Ascend310P48: "1"
          huawei.com/Ascend310P48-memory: "8000"

When you pick an Ascend GPU type on the workload create page, enter memory in MiB from the table above and leave core empty. See Container Image Settings.

Small templates and graph capture

16G / 32G hard templates have fewer AI Cores and streams than a whole card. Default vLLM-Ascend ACL Graph capture can fail with EE1023 (stream resources exhausted). Use one of:

• Add --enforce-eager to disable graph capture
• Reduce cudagraph_capture_sizes, or move to a larger template / whole card

The Pod detail page Scheduled to GPU section shows the result. A hard slice lists the template name (training: vir12_3c_32g; inference: vir10_3c_32g) and memory, not a core percentage. Cluster-side views are in GPU Card Management.