Every day Kubernetes
In this article I will post kubectl commands and Kubernetes tricks I use as part of my daily routine. I don't remember things by heart, but I am also sick
and tired of googling each time. So here is the list of absolute must-know CLI commands that is used to operate a cluster on daily basis.
The kubectl CLI command name is objectively challenging to spell. The letter combination is not handy and can't be easily typed in. So, I define an alias:
alias k="kubectl"
The next thing I may want to do is to see the list of contexts. Each context represents a separate cluster. To see the list of contexts:
k config get-contexts
To switch to a specific context:
k config use-context <CONTEXT_NAME>
To delete a context no longer needed:
k config delete-context <CONTEXT_NAME>
If find it useful to add some aliasing for the first two commands:
alias kctx="k config get-contexts"alias kuctx="k config use-context"
The way to add new contexts depends on the kind of cloud provider used. For GCP the method is the following:
- Install and setup gcloud cli
- Activate your project:
gcloud config set project <PROJECT_ID> - Knowing the cluster name and the zone it operates in, add the context:
gcloud container clusters get-credentials <CLUSTER_NAME> --zone <CLUSTER_ZONE>
To see the list of all namespaces on a cluster:
k get namespace
As most of the time I work with just one application, it makes sense narrow the scope to just one particular namespace:
alias kn="k -n <NAMESPACE_OF_MY_APP>"
To see the list of pods, there is a command:
kn get pod
To search for pods across all namespaces (useful when the namespace is unknown):
kn get pods --all-namespaces | grep <POD_NAME>
To see the list of pods and containers running inside each of them, here is a snippet:
kn get pods -o jsonpath='{range .items[]}{"\n"}{.metadata.name}{"\t"}{range .spec.containers[]}{.name}{"=>"}{.image}{"\t"}{end}{end}'|sort|column -t
To get a list of containers and images they are spinning, here is another snippet:
kn get pods -o json | jq '.items[] | { "name": .metadata.labels.name, "country": .metadata.labels.country, "image": .status.containerStatuses[].imageID }'
Note, that I select imageID, not image, because the sha digest is much more valuable than just an image name + a tag.
Sometimes it is needed to read the job log to see why it failed. Every job is backed by a temporary pod. Knowing the pod name, we can get to the logs and hopefully understand the reason.
To get the pod name that backs the job, use:
kn get pods --selector=job-name=<JOB_NAME>
There is a way to see the detailed information about a pod: labels, a list of containers, etc.
kn get pods <POD_NAME> -o json
kn get pods <POD_NAME> -o jsonpath='{.spec.containers[*].name}'
To see the logs there is a command:
kn logs <POD_NAME> -c <CONTAINER_NAME>
If you don't specify the -c parameter, it will print logs of the default container of the pod (this is probably not what you want, so get the list of containers on the pod first).
If there is a job you are running inside of the cluster, and you need real time log output, there is the "-f" option:
kn logs -f <POD_NAME> -c <CONTAINER_NAME>
Exec into a container may be last resort. But, anyway, there is how it is done:
kn exec -it <POD_NAME> -c <CONTAINER_NAME> -- sh
Most of the time I use it when I need to check the env vars with printenv.
Sometimes I want a pod restarted, in case if it misbehaves. I scale it down to zero and then up again.
kn scale deploy <POD_NAME> --replicas=0kn scale deploy <POD_NAME> --replicas=<ORIGINAL_NUM_OF_REPLICAS>
If a deployment is reported to be scaled up, but nothing shows up, it's a good case for reading the event log of the K8s itself:
kn get events
If I have a regular job, or a cron job, I may want to see how it is doing. Assuming that I have the jq tool installed, I can then type:
kn get job -l <LABEL_NAME>=<LABEL_VALUE> -o json | jq -r '.items[] | select(.status.succeeded) | .status.completionTime' | sort -r | head -n 1
The -l command is a selector by label. It accepts a label name and label value to filter the list of jobs against.
Sometimes an environment variable must be injected into a working pod, without changing the terraform files or redeploying. For example, if there is a need to temporary increase the logging level from "warn" to "debug". This is how it's done:
kn set env deployment/<DEPLOYMENT_NAME> <ENV_VAR_NAME>=<VALUE>
This command will patch the deployment and do the rolling update.
Most of the time, there is no connectivity between a container and an outer world, because a port may not be exposed via an ingress. If debugging is needed, it is possible to do port forwarding to a local machine.
kn port-forward pod/<POD_NAME> <LOCAL_PORT>:<POD_PORT>
After running this command, the pod becomes available through localhost:<LOCAL_PORT>.
You can also connect to the service/deployment, and then K8s will make use of the internal load balancer if the deployment is scaled:
kn port-forward service/<SERVICE_NAME> <LOCAL_PORT>:<POD_PORT>
Or, for a deployment:
kn port-forward deployment/<DEPLOYMENT_NAME> <LOCAL_PORT>:<POD_PORT>
Sometimes, just for the sake of debugging, I may want to create a new container to check/debug something. I can then pick an image, run it and SSH into it. After the work is done, the container gets removed.
kn run -i --tty my-test-container --image=ubuntu --restart=Never --rm -- /bin/bash
If a cluster resource was deleted by mistake, and you have a yaml file describing it (most likely in the infra repository),
no need to worry, as it can be easily restored using the following command:
kn apply -f resources/my-missing-resource.yaml
Don't forget to apply those:
source ~/.bash_profile
As usual, this is a work in progress article. I will add more commands later as soon as I have them!
