Taints and Tolerance in kubernetes

Taints restrict pods that can be scheduled on a node. They are assigned on the node as key value pair, Such that only nodes which can tolerate the taint can be scheduled on the Nodes.

Tolerance is assigned to a pod so that is can tolerate a certain taint on a Node. They have a key value pair to suggest what type of taint to tolerate.

Tolerance also have effect which specify the type of scheduling, there are 3 effect

• NoSchedule - Applyies to new pods

• PreferNoSchedule - Could apply to new pods with no guarantee

• NoExecuting - Apply to new/existing pods

This is the command to Taint a node.

kubectl taint node <node-name> key=value:NoSchedule

run this for all the worker-nodes

kubectl taint node <worker-node> gpu=true:NoSchedule

gpu is the taint key and true is the value.

Now create a yaml config for a pod with tolerance to the taint

kubectl run redis —image=redis —dry-run=client -o yaml > redis.yaml

Edit the file by adding tolerance

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: redis
  name: redis
spec:
  containers:
    - image: redis
      name: redis
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
status: {}

For the first time comment the tolerance attribute and create the pod

kubectl create -f redis.yaml

Then Check pod Status

kubectl get po

the status should be pending if all worker-nodes were tainted

Now delete the pod

kubectl delete pod redis

Then uncomment the tolerance attribute and re-create the pod

kubectl create -f redis.yaml

Check the status. Now it should run cause it can tolarate the taint.

Labels and NodeSelectors

Taints and tolerance prevent unwanted pods from being sheduled on a Node but do not guarantee that the pods with the tolerance will be sheduled on the Node with the taint. In this case the Node filter the pods to shedule.

But if you want to shedule pods on certain specific node, you use labels and nodeSelector to match the pods to a labeled Node.

To label a node run

kubectl label node <node-name> gpu=false

Do this to one worker in the cluster.

Then write a pod config with a node selector that matches a single Node in the cluster, or multiple if they got the same label

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx-new
  name: nginx-new
spec:
  containers:
    - image: nginx
      name: nginx-new
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  nodeSelector:
    gpu: "false"
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
status: {}

The nodeSelector field matches the podto nodes with only that label. That the pod can determine where it will be schedules.