kubectl Commands for Day-to-Day Operations

Posted on Sun, Nov 5, 2023
Tags kubernetes, linux
πŸ“ This article is a translation of the original Japanese post. View original

A reference collection of kubectl commands and tips.

1# Create an alias so you don't have to type kubectl every time
2alias k=kubectl

Get Pod Information

1# all namespaces
2$ kubectl get pods -A
3NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
4kube-system   coredns-5dd5756b68-ff4ql                 1/1     Running   0          41s
5kube-system   coredns-5dd5756b68-jpl5z                 1/1     Running   0          41s
6kube-system   etcd-docker-desktop                      1/1     Running   0          46s

Getting Information

kubectl get all does not list all resources. Use api-resources to enumerate API types

Reference: kubectl get all doesn’t show all resources | text.superbrothers.dev

1kubectl get -A "$(kubectl api-resources --namespaced=true --verbs=list --output=name | tr "\n" "," | sed -e 's/,$//')"

Use kubectl api-resources to create a CSV of resource names, then pass them to kubectl get "x,y,z,...".

View kubectl get events chronologically

Use --sort-by for chronological order and set -o output format to go-template for TSV output.

1$ kubectl get events -A --sort-by='.metadata.creationTimestamp' -o 'go-template={{range .items}}{{.metadata.creationTimestamp}}{{"\t"}}{{.type}}{{"\t"}}{{.involvedObject.kind}}{{"\t"}}{{.reason}}{{"\t"}}{{.involvedObject.name}}{{"\t"}}{{.message}}{{"\n"}}{{end}}'
2
32023-01-01T07:22:33Z    Normal  Node    NodeHasNoDiskPressure   kind-control-plane      Node kind-control-plane status is now: NodeHasNoDiskPressure
42023-01-01T07:22:33Z    Normal  Node    NodeHasSufficientPID    kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientPID
52023-01-01T07:22:33Z    Normal  Node    NodeHasSufficientMemory kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientMemory
62023-01-01T07:22:34Z    Normal  Node    Starting        kind-control-plane      Starting kubelet.
72023-01-01T07:22:34Z    Normal  Node    NodeHasSufficientMemory kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientMemory
82023-01-01T07:22:34Z    Normal  Node    NodeHasNoDiskPressure   kind-control-plane      Node kind-control-plane status is now: NodeHasNoDiskPressure
92023-01-01T07:22:34Z    Normal  Lease   LeaderElection  kube-controller-manager kind-control-plane_bfba68b7-7aa5-4746-9c6b-0addb52ea429 became leader

Use field selectors as well as label selectors

When filtering by pod or node type, label selectors (-l) are often used, but field selectors allow filtering by status information. In most cases you can get by with grep, but field selectors are useful when you want output in json or yaml format.

1$ kubectl get pods -A --field-selector status.phase=Running
2
3NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
4kube-system          coredns-565d847f94-5xxhc                     1/1     Running   0          74m
5kube-system          coredns-565d847f94-k7tpv                     1/1     Running   0          74m

Extract only the information you need with -o jsonpath

This is used frequently, so there are many examples on the internet. Since the contents of spec differ by environment, first check with -oyaml, then extract with jsonpath.

 1# Get the name of the node a pod is running on
 2$ kubectl get pod -nkube-system kube-apiserver-kind-control-plane -o jsonpath='{.spec.nodeName}'
 3kind-control-plane
 4
 5# Control plane IP
 6$ kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'
 7172.18.0.2
 8
 9# You can also use jsonpath range to format output
10$ kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}'
11172.18.0.2

Combine custom columns with the read command to process in bash

Useful for operations during incidents where you need to act on all nodes or pods.

 1$ while read -r ns pod node
 2do
 3    echo "processing... $ns, $pod, $node"
 4done < <(kubectl get pod --no-headers -A -ocustom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName)
 5
 6processing... kube-system, coredns-565d847f94-5xxhc, kind-control-plane
 7processing... kube-system, coredns-565d847f94-k7tpv, kind-control-plane
 8processing... kube-system, etcd-kind-control-plane, kind-control-plane
 9processing... kube-system, kindnet-pgflw, kind-control-plane
10processing... kube-system, kube-apiserver-kind-control-plane, kind-control-plane
11processing... kube-system, kube-controller-manager-kind-control-plane, kind-control-plane
12processing... kube-system, kube-proxy-7g4tk, kind-control-plane
13processing... kube-system, kube-scheduler-kind-control-plane, kind-control-plane
14processing... local-path-storage, local-path-provisioner-684f458cdd-fdfzb, kind-control-plane

Check base64-encoded secrets

The option varies slightly depending on the Linux or bash version, but the following lets you receive a base64-encoded string and decode it.

1echo -n "base64 password: "; read -s pswd; echo "$pswd" | base64 --decode | less

If there’s a specific secret, use -o jsonpath cleverly. The following is an example of base64 decoding an argocd secret value.

1kubectl get secret -nargocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 --decode | less

Troubleshooting

Dump cluster information

1kubectl cluster-info dump

Force-delete a misbehaving pod immediately

Forcefully deleting a pod means finalizers won’t clean up garbage resources, and the app receives SIGKILL instead of SIGTERM, preventing graceful shutdown. You need to understand this before using --force.

1kubectl delete pod --grace-period=0 --force --wait=false

kubectl debug

Without specifying --target, you can’t mount the target container’s process.

 1$ kubectl debug -nnamespace -it --image=ubuntu:latest mysql-7d48796987-qbtww --target=mysql -- bash
 2
 3# The target pod's volume exists under /proc/<pid>/root
 4root@mysql-7d48796987-74tsw:/# ps aux
 5USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 6root           1  0.0  0.0   4624  3888 pts/0    Ss   14:03   0:00 bash
 7root          17  0.0  0.0   7060  1608 pts/0    R+   14:06   0:00 ps aux
 8
 9# Not obvious, but you can see the target volume inside /root under the process
10root@mysql-7d48796987-74tsw:/# ls /proc/1/root/
11bin   dev  home  lib32  libx32  mnt  proc          root  sbin  sys  usr
12boot  etc  lib   lib64  media   opt  product_uuid  run   srv   tmp  var
13
14root@mysql-7d48796987-74tsw:/# ls /proc/1/root/tmp
15mysql.sock
16
17# If you want to name the debugger container, specify with -c. Without it (as above), debug containers are created with hash-suffixed names
18$ kubectl debug -nnamespace -it --image=ubuntu:latest mysql-7d48796987-qbtww --target=mysql -c=debugger -- bash
19
20# If the pod is managed by a deployment, you can delete the debug container along with the pod by deleting the pod
21# We'd love --rm option support
22$ kubectl delete pod mysql-7d48796987-qbtww

About kubectl krew

Installing krew plugins lets you make commands even more convenient: Kubectl plugins available Β· Krew

tree

1$ kubectl krew install tree
2
3$ kubectl tree deployment grafana -nproduct-measurement
4NAMESPACE            NAME                             READY  REASON  AGE
5product-measurement  Deployment/grafana               -              21h
6product-measurement  └─ReplicaSet/grafana-96dd49547   -              21h
7product-measurement    └─Pod/grafana-96dd49547-49czh  True           21h

ahmetb/kubectl-tree: kubectl plugin to browse Kubernetes object hierarchies as a tree πŸŽ„

Lets you see resource dependency relationships in a tree structure, which is easy to understand.

neat

itaysk/kubectl-neat: Clean up Kubernetes yaml and json output to make it readable

A plugin that filters and displays yaml, removing unnecessary information.

 1# no last applied configuration etc.
 2$ kubectl neat get -- deployment grafana -nproduct-measurement
 3apiVersion: apps/v1
 4kind: Deployment
 5metadata:
 6  annotations:
 7    deployment.kubernetes.io/revision: "1"
 8  labels:
 9    app: grafana
10  name: grafana
11  namespace: product-measurement
12spec:
13  progressDeadlineSeconds: 600
14  replicas: 1
15  revisionHistoryLimit: 10
16  selector:
17    matchLabels:
18      app: grafana

access-matrix

 1$ kubectl access-matrix
 2NAME                                                          LIST  CREATE  UPDATE  DELETE
 3apiservices.apiregistration.k8s.io                            βœ”     βœ”       βœ”       βœ”
 4bindings                                                            βœ”
 5certificatesigningrequests.certificates.k8s.io                βœ”     βœ”       βœ”       βœ”
 6clusterrolebindings.rbac.authorization.k8s.io                 βœ”     βœ”       βœ”       βœ”
 7clusterroles.rbac.authorization.k8s.io                        βœ”     βœ”       βœ”       βœ”
 8componentstatuses                                             βœ”
 9configmaps                                                    βœ”     βœ”       βœ”       βœ”
10controllerrevisions.apps                                      βœ”     βœ”       βœ”       βœ”
11cronjobs.batch                                                βœ”     βœ”       βœ”       βœ”
12csidrivers.storage.k8s.io                                     βœ”     βœ”       βœ”       βœ”
13csinodes.storage.k8s.io                                       βœ”     βœ”       βœ”       βœ”
14csistoragecapacities.storage.k8s.io                           βœ”     βœ”       βœ”       βœ”
15customresourcedefinitions.apiextensions.k8s.io                βœ”     βœ”       βœ”       βœ”
16daemonsets.apps                                               βœ”     βœ”       βœ”       βœ”
17deployments.apps                                              βœ”     βœ”       βœ”       βœ”
18endpoints                                                     βœ”     βœ”       βœ”       βœ”
19endpointslices.discovery.k8s.io                               βœ”     βœ”       βœ”       βœ”
20events                                                        βœ”     βœ”       βœ”       βœ”
21events.events.k8s.io                                          βœ”     βœ”       βœ”       βœ”
22flowschemas.flowcontrol.apiserver.k8s.io                      βœ”     βœ”       βœ”       βœ”
23horizontalpodautoscalers.autoscaling                          βœ”     βœ”       βœ”       βœ”
24ingressclasses.networking.k8s.io                              βœ”     βœ”       βœ”       βœ”

rolesum

Lets you investigate permissions by account.

1$ kubectl rolesum -n kube-system bootstrap-signer
2ServiceAccount: kube-system/bootstrap-signer
3Secrets:
4
5Policies:
6β€’ [RB] kube-system/system:controller:bootstrap-signer ⟢  [R] kube-system/system:controller:bootstrap-signer
7  Resource  Name  Exclude  Verbs  G L W C U P D DC
8  secrets   [*]     [-]     [-]   βœ” βœ” βœ” βœ– βœ– βœ– βœ– βœ–

ctx, ns

1$ k ctx
2
3$ k ns
4Context "kind-kind" modified.
5Active namespace is "kube-system".

Share

See also

Edit on GitHub