Skip to content

Node labels and taints

A label is a key=value pair on a node that pods can select it by. A taint keeps pods off a node unless they tolerate it. Together they dedicate a pool to a workload: label a GPU pool so GPU jobs choose it, and taint it so nothing else lands there.

In Hyperkub you set labels and taints on a node pool, and every node in the pool gets them, including nodes added later by a resize.

Hyperkub labels every node with where it belongs, so you can select a pool without setting anything yourself:

Label Value
hyperkub.io/node-pool-id The node pool’s ID.
hyperkub.io/node-pool-name The node pool’s name.
hyperkub.io/size The node size.
hyperkub.io/version The Kubernetes version the node runs.

Kubernetes adds its own as well, such as kubernetes.io/hostname.

Pass labels and taints when you create the pool, or in a pool of a new cluster’s pools:

Terminal window
curl -X POST "https://api.hyperkub.com/v1/cluster_pools" \
-H "Authorization: Bearer $HYPERKUB_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"cluster_id": "01j…",
"name": "gpu",
"size": "<size>",
"count": 2,
"labels": { "workload": "gpu" },
"taints": [{ "key": "dedicated", "value": "gpu", "effect": "NoSchedule" }]
}'

labels is an object of string values. Each taint has a key, an optional value and an effect:

Effect Pods that do not tolerate it
NoSchedule Are not scheduled on the node. Pods already there stay.
PreferNoSchedule Are scheduled elsewhere if possible.
NoExecute Are not scheduled, and those already running are evicted.

Send the new labels, taints or both to the pool:

Terminal window
curl -X PATCH "https://api.hyperkub.com/v1/cluster_pools/$POOL_ID" \
-H "Authorization: Bearer $HYPERKUB_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "taints": [{ "key": "dedicated", "value": "gpu", "effect": "NoExecute" }] }'
  • What you send replaces the whole set. It is not merged with what the pool has. To add one label, send the existing ones too. An empty {} or [] removes them all.
  • What you leave out stays as it is. The request above changes the taints and keeps the labels.
  • Every node is updated. The pool and its nodes show updating until all are done, then running again. The change appears in the account’s activity as one entry, with a step per node.
  • Labels and taints set with kubectl are removed. Each node ends up with exactly the pool’s set and Hyperkub’s labels. Labels and taints Kubernetes sets itself, in the kubernetes.io and k8s.io namespaces, are kept.

A change needs every node in the pool to be running, and cannot be combined with a change to capacity in the same request. Sending what the pool already has does nothing.

  • Keys follow Kubernetes’ rules: an optional DNS-subdomain prefix and a /, then a name of up to 63 letters, digits, -, _ and ., such as workload or example.com/team. Values follow the same rules for names, and can be empty.
  • Keys in the kubernetes.io, k8s.io and hyperkub.io namespaces, and their subdomains, are refused. They belong to Kubernetes and Hyperkub.
  • A pool cannot have two taints with the same key and effect.
  • A cluster’s control-plane pool has no labels or taints of yours.

Select the pool by its label and tolerate its taint:

apiVersion: apps/v1
kind: Deployment
metadata:
name: trainer
spec:
replicas: 1
selector:
matchLabels: { app: trainer }
template:
metadata:
labels: { app: trainer }
spec:
nodeSelector:
workload: gpu
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
containers:
- name: trainer
image: example/trainer:1.0

To select a pool by name instead, use hyperkub.io/node-pool-name: gpu as the nodeSelector.

Check what a node has with kubectl:

Terminal window
kubectl get nodes -L workload,hyperkub.io/node-pool-name
kubectl describe node <node> | grep -A3 Taints

AI agents can change a pool’s labels and taints with the update_node_pool_labels_and_taints tool.