kubectl Commands for Day-to-Day Operations

📝 This article is a translation of the original Japanese post. View original

A reference collection of kubectl commands and tips.

# Create an alias so you don't have to type kubectl every time
alias k=kubectl

Get Pod Information

# all namespaces
$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-ff4ql                 1/1     Running   0          41s
kube-system   coredns-5dd5756b68-jpl5z                 1/1     Running   0          41s
kube-system   etcd-docker-desktop                      1/1     Running   0          46s

Getting Information

`kubectl get all` does not list all resources. Use api-resources to enumerate API types

Reference: kubectl get all doesn’t show all resources | text.superbrothers.dev

kubectl get -A "$(kubectl api-resources --namespaced=true --verbs=list --output=name | tr "\n" "," | sed -e 's/,$//')"

Use kubectl api-resources to create a CSV of resource names, then pass them to kubectl get "x,y,z,...".

View kubectl get events chronologically

Use --sort-by for chronological order and set -o output format to go-template for TSV output.

$ kubectl get events -A --sort-by='.metadata.creationTimestamp' -o 'go-template={{range .items}}{{.metadata.creationTimestamp}}{{"\t"}}{{.type}}{{"\t"}}{{.involvedObject.kind}}{{"\t"}}{{.reason}}{{"\t"}}{{.involvedObject.name}}{{"\t"}}{{.message}}{{"\n"}}{{end}}'

2023-01-01T07:22:33Z    Normal  Node    NodeHasNoDiskPressure   kind-control-plane      Node kind-control-plane status is now: NodeHasNoDiskPressure
2023-01-01T07:22:33Z    Normal  Node    NodeHasSufficientPID    kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientPID
2023-01-01T07:22:33Z    Normal  Node    NodeHasSufficientMemory kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientMemory
2023-01-01T07:22:34Z    Normal  Node    Starting        kind-control-plane      Starting kubelet.
2023-01-01T07:22:34Z    Normal  Node    NodeHasSufficientMemory kind-control-plane      Node kind-control-plane status is now: NodeHasSufficientMemory
2023-01-01T07:22:34Z    Normal  Node    NodeHasNoDiskPressure   kind-control-plane      Node kind-control-plane status is now: NodeHasNoDiskPressure
2023-01-01T07:22:34Z    Normal  Lease   LeaderElection  kube-controller-manager kind-control-plane_bfba68b7-7aa5-4746-9c6b-0addb52ea429 became leader

Use field selectors as well as label selectors

When filtering by pod or node type, label selectors (-l) are often used, but field selectors allow filtering by status information. In most cases you can get by with grep, but field selectors are useful when you want output in json or yaml format.

$ kubectl get pods -A --field-selector status.phase=Running

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-565d847f94-5xxhc                     1/1     Running   0          74m
kube-system          coredns-565d847f94-k7tpv                     1/1     Running   0          74m

Extract only the information you need with -o jsonpath

This is used frequently, so there are many examples on the internet. Since the contents of spec differ by environment, first check with -oyaml, then extract with jsonpath.

# Get the name of the node a pod is running on
$ kubectl get pod -nkube-system kube-apiserver-kind-control-plane -o jsonpath='{.spec.nodeName}'
kind-control-plane

# Control plane IP
$ kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'
172.18.0.2

# You can also use jsonpath range to format output
$ kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}'
172.18.0.2

Combine custom columns with the read command to process in bash

Useful for operations during incidents where you need to act on all nodes or pods.

$ while read -r ns pod node
do
    echo "processing... $ns, $pod, $node"
done < <(kubectl get pod --no-headers -A -ocustom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName)

processing... kube-system, coredns-565d847f94-5xxhc, kind-control-plane
processing... kube-system, coredns-565d847f94-k7tpv, kind-control-plane
processing... kube-system, etcd-kind-control-plane, kind-control-plane
processing... kube-system, kindnet-pgflw, kind-control-plane
processing... kube-system, kube-apiserver-kind-control-plane, kind-control-plane
processing... kube-system, kube-controller-manager-kind-control-plane, kind-control-plane
processing... kube-system, kube-proxy-7g4tk, kind-control-plane
processing... kube-system, kube-scheduler-kind-control-plane, kind-control-plane
processing... local-path-storage, local-path-provisioner-684f458cdd-fdfzb, kind-control-plane

Check base64-encoded secrets

The option varies slightly depending on the Linux or bash version, but the following lets you receive a base64-encoded string and decode it.

echo -n "base64 password: "; read -s pswd; echo "$pswd" | base64 --decode | less

If there’s a specific secret, use -o jsonpath cleverly. The following is an example of base64 decoding an argocd secret value.

kubectl get secret -nargocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 --decode | less

Troubleshooting

Dump cluster information

kubectl cluster-info dump

Force-delete a misbehaving pod immediately

Forcefully deleting a pod means finalizers won’t clean up garbage resources, and the app receives SIGKILL instead of SIGTERM, preventing graceful shutdown. You need to understand this before using --force.

kubectl delete pod --grace-period=0 --force --wait=false

kubectl debug

Without specifying --target, you can’t mount the target container’s process.

$ kubectl debug -nnamespace -it --image=ubuntu:latest mysql-7d48796987-qbtww --target=mysql -- bash

# The target pod's volume exists under /proc/<pid>/root
root@mysql-7d48796987-74tsw:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   4624  3888 pts/0    Ss   14:03   0:00 bash
root          17  0.0  0.0   7060  1608 pts/0    R+   14:06   0:00 ps aux

# Not obvious, but you can see the target volume inside /root under the process
root@mysql-7d48796987-74tsw:/# ls /proc/1/root/
bin   dev  home  lib32  libx32  mnt  proc          root  sbin  sys  usr
boot  etc  lib   lib64  media   opt  product_uuid  run   srv   tmp  var

root@mysql-7d48796987-74tsw:/# ls /proc/1/root/tmp
mysql.sock

# If you want to name the debugger container, specify with -c. Without it (as above), debug containers are created with hash-suffixed names
$ kubectl debug -nnamespace -it --image=ubuntu:latest mysql-7d48796987-qbtww --target=mysql -c=debugger -- bash

# If the pod is managed by a deployment, you can delete the debug container along with the pod by deleting the pod
# We'd love --rm option support
$ kubectl delete pod mysql-7d48796987-qbtww

About kubectl krew

Installing krew plugins lets you make commands even more convenient: Kubectl plugins available · Krew

tree

$ kubectl krew install tree

$ kubectl tree deployment grafana -nproduct-measurement
NAMESPACE            NAME                             READY  REASON  AGE
product-measurement  Deployment/grafana               -              21h
product-measurement  └─ReplicaSet/grafana-96dd49547   -              21h
product-measurement    └─Pod/grafana-96dd49547-49czh  True           21h

ahmetb/kubectl-tree: kubectl plugin to browse Kubernetes object hierarchies as a tree 🎄

Lets you see resource dependency relationships in a tree structure, which is easy to understand.

neat

itaysk/kubectl-neat: Clean up Kubernetes yaml and json output to make it readable

A plugin that filters and displays yaml, removing unnecessary information.

# no last applied configuration etc.
$ kubectl neat get -- deployment grafana -nproduct-measurement
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: grafana
  name: grafana
  namespace: product-measurement
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: grafana

access-matrix

$ kubectl access-matrix
NAME                                                          LIST  CREATE  UPDATE  DELETE
apiservices.apiregistration.k8s.io                            ✔     ✔       ✔       ✔
bindings                                                            ✔
certificatesigningrequests.certificates.k8s.io                ✔     ✔       ✔       ✔
clusterrolebindings.rbac.authorization.k8s.io                 ✔     ✔       ✔       ✔
clusterroles.rbac.authorization.k8s.io                        ✔     ✔       ✔       ✔
componentstatuses                                             ✔
configmaps                                                    ✔     ✔       ✔       ✔
controllerrevisions.apps                                      ✔     ✔       ✔       ✔
cronjobs.batch                                                ✔     ✔       ✔       ✔
csidrivers.storage.k8s.io                                     ✔     ✔       ✔       ✔
csinodes.storage.k8s.io                                       ✔     ✔       ✔       ✔
csistoragecapacities.storage.k8s.io                           ✔     ✔       ✔       ✔
customresourcedefinitions.apiextensions.k8s.io                ✔     ✔       ✔       ✔
daemonsets.apps                                               ✔     ✔       ✔       ✔
deployments.apps                                              ✔     ✔       ✔       ✔
endpoints                                                     ✔     ✔       ✔       ✔
endpointslices.discovery.k8s.io                               ✔     ✔       ✔       ✔
events                                                        ✔     ✔       ✔       ✔
events.events.k8s.io                                          ✔     ✔       ✔       ✔
flowschemas.flowcontrol.apiserver.k8s.io                      ✔     ✔       ✔       ✔
horizontalpodautoscalers.autoscaling                          ✔     ✔       ✔       ✔
ingressclasses.networking.k8s.io                              ✔     ✔       ✔       ✔

rolesum

Lets you investigate permissions by account.

$ kubectl rolesum -n kube-system bootstrap-signer
ServiceAccount: kube-system/bootstrap-signer
Secrets:

Policies:
• [RB] kube-system/system:controller:bootstrap-signer ⟶  [R] kube-system/system:controller:bootstrap-signer
  Resource  Name  Exclude  Verbs  G L W C U P D DC
  secrets   [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖

ctx, ns

$ k ctx

$ k ns
Context "kind-kind" modified.
Active namespace is "kube-system".

Get Pod Information

Getting Information

kubectl get all does not list all resources. Use api-resources to enumerate API types

View kubectl get events chronologically

Use field selectors as well as label selectors

Extract only the information you need with -o jsonpath

Combine custom columns with the read command to process in bash

Check base64-encoded secrets

Troubleshooting

Dump cluster information

Force-delete a misbehaving pod immediately

kubectl debug

About kubectl krew

tree

neat

access-matrix

rolesum

ctx, ns

Share with

`kubectl get all` does not list all resources. Use api-resources to enumerate API types