Node labels and taints
A label is a key=value pair on a node that pods can select it by. A
taint keeps pods off a node unless they tolerate it. Together they
dedicate a pool to a workload: label a GPU pool so GPU jobs choose it, and taint
it so nothing else lands there.
In Hyperkub you set labels and taints on a node pool, and every node in the pool gets them, including nodes added later by a resize.
Labels every node has
Section titled “Labels every node has”Hyperkub labels every node with where it belongs, so you can select a pool without setting anything yourself:
| Label | Value |
|---|---|
hyperkub.io/node-pool-id |
The node pool’s ID. |
hyperkub.io/node-pool-name |
The node pool’s name. |
hyperkub.io/size |
The node size. |
hyperkub.io/version |
The Kubernetes version the node runs. |
Kubernetes adds its own as well, such as kubernetes.io/hostname.
Set them on a new pool
Section titled “Set them on a new pool”Pass labels and taints when you create the pool, or in a pool of a new
cluster’s pools:
curl -X POST "https://api.hyperkub.com/v1/cluster_pools" \ -H "Authorization: Bearer $HYPERKUB_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cluster_id": "01j…", "name": "gpu", "size": "<size>", "count": 2, "labels": { "workload": "gpu" }, "taints": [{ "key": "dedicated", "value": "gpu", "effect": "NoSchedule" }] }'labels is an object of string values. Each taint has a key, an optional
value and an effect:
| Effect | Pods that do not tolerate it |
|---|---|
NoSchedule |
Are not scheduled on the node. Pods already there stay. |
PreferNoSchedule |
Are scheduled elsewhere if possible. |
NoExecute |
Are not scheduled, and those already running are evicted. |
Change them on an existing pool
Section titled “Change them on an existing pool”Send the new labels, taints or both to the pool:
curl -X PATCH "https://api.hyperkub.com/v1/cluster_pools/$POOL_ID" \ -H "Authorization: Bearer $HYPERKUB_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "taints": [{ "key": "dedicated", "value": "gpu", "effect": "NoExecute" }] }'- What you send replaces the whole set. It is not merged with what the pool
has. To add one label, send the existing ones too. An empty
{}or[]removes them all. - What you leave out stays as it is. The request above changes the taints and keeps the labels.
- Every node is updated. The pool and its nodes show
updatinguntil all are done, thenrunningagain. The change appears in the account’s activity as one entry, with a step per node. - Labels and taints set with
kubectlare removed. Each node ends up with exactly the pool’s set and Hyperkub’s labels. Labels and taints Kubernetes sets itself, in thekubernetes.ioandk8s.ionamespaces, are kept.
A change needs every node in the pool to be running, and cannot be combined
with a change to capacity in the same request. Sending what the pool already
has does nothing.
- Keys follow Kubernetes’ rules: an optional DNS-subdomain prefix and a
/, then a name of up to 63 letters, digits,-,_and., such asworkloadorexample.com/team. Values follow the same rules for names, and can be empty. - Keys in the
kubernetes.io,k8s.ioandhyperkub.ionamespaces, and their subdomains, are refused. They belong to Kubernetes and Hyperkub. - A pool cannot have two taints with the same key and effect.
- A cluster’s control-plane pool has no labels or taints of yours.
Use them in a workload
Section titled “Use them in a workload”Select the pool by its label and tolerate its taint:
apiVersion: apps/v1kind: Deploymentmetadata: name: trainerspec: replicas: 1 selector: matchLabels: { app: trainer } template: metadata: labels: { app: trainer } spec: nodeSelector: workload: gpu tolerations: - key: dedicated operator: Equal value: gpu effect: NoSchedule containers: - name: trainer image: example/trainer:1.0To select a pool by name instead, use
hyperkub.io/node-pool-name: gpu as the nodeSelector.
Check what a node has with kubectl:
kubectl get nodes -L workload,hyperkub.io/node-pool-namekubectl describe node <node> | grep -A3 TaintsAI agents can change a pool’s labels and taints with the
update_node_pool_labels_and_taints
tool.