The commands that aren't in any manual

Everything you've used so far is in the manuals. This lesson goes through the thirteen commands that aren't: the ones you learn by looking over the shoulder of someone who has been operating clusters for years. Five of them come with a mission; the rest, read slowly, because the day you need them there won't be time to look them up.

The scenario: a Pod called objetivo (Spanish for "target") awaiting its fate, and a freshly created ServiceAccount worker. Work from the dev-machine tab.

Mission 1: an ephemeral token

An external process needs to authenticate as the worker ServiceAccount for the next 10 minutes, not one more. The old way was to create a long-lived Secret; the modern way (since Kubernetes 1.24) doesn't touch Secrets:

How do I generate a short-lived token for a ServiceAccount?

kubectl create token worker --duration=10m > /home/laborant/token.txt

Open it: it's a JWT (three base64 blocks separated by dots) with a built-in expiry. It ties straight back to the Security module: an ephemeral identity instead of an eternal credential.

And now what do I do with it? Use it. A token is a credential: you present it in the Authorization header of any request to the API. Ask the cluster who you are when you're wearing it:

SERVIDOR=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

curl -sk -H "Authorization: Bearer $(cat /home/laborant/token.txt)" "${SERVIDOR}/apis/authentication.k8s.io/v1/selfsubjectreviews" -X POST -H 'Content-Type: application/json' -d '{"apiVersion":"authentication.k8s.io/v1","kind":"SelfSubjectReview"}' | jq .status.userInfo

The API answers with the identity it recognized: system:serviceaccount:default:worker, its groups and the expiry. The token works.

Now ask it for something real:

curl -sk -H "Authorization: Bearer $(cat /home/laborant/token.txt)" "${SERVIDOR}/api/v1/namespaces/default/pods" | jq -r '.message // .kind'

pods is forbidden. And there's the lesson that trips up a lot of people: the token says who you are, not what you can do. Authentication and authorization are two different doors, and you've only crossed the first. Give that identity read permission and repeat the call:

kubectl create rolebinding worker-lector --clusterrole=view --serviceaccount=default:worker

curl -sk -H "Authorization: Bearer $(cat /home/laborant/token.txt)" "${SERVIDOR}/api/v1/namespaces/default/pods" | jq -r '.items[].metadata.name'

The list of Pods, with the same token as before and without restarting anything. What changed wasn't the credential: it was the RoleBinding.

The -k in the curl skips validation of the API server's certificate. In the lab it's fine; in production you pass the CA with --cacert, as you did from inside a Pod in the Security module.

Mission 2: a pocket kubeconfig

You have to hand access to this cluster to an external tool, and your ~/.kube/config may accumulate contexts for several clusters and references to certificate files the tool won't have.

How do I generate a clean, self-contained kubeconfig with only the active context?

kubectl config view --minify --flatten > /home/laborant/kubeconfig-lab.yaml

--minify trims everything that isn't the current context; --flatten embeds the certificates inside the file itself. The result works anywhere as-is.

But notice what you've just put into that file: your administrator credentials. Handing it over like that is handing over the keys to the whole cluster. Swap them for the token from the previous mission, which expires in ten minutes and can only read:

kubectl --kubeconfig=/home/laborant/kubeconfig-lab.yaml config set-credentials worker --token="$(cat /home/laborant/token.txt)"

kubectl --kubeconfig=/home/laborant/kubeconfig-lab.yaml config set-context --current --user=worker

And now use it, which is the proof that it's good for something:

kubectl --kubeconfig=/home/laborant/kubeconfig-lab.yaml auth whoami

kubectl --kubeconfig=/home/laborant/kubeconfig-lab.yaml get pods

The first answers system:serviceaccount:default:worker: kubectl stops being you and becomes the ServiceAccount. The second lists the Pods, because you gave it view a moment ago. Try deleting something with that kubeconfig and you'll see the Forbidden from the other side of the counter.

That's what you hand to an external tool, a CI runner or a colleague: one file, a bounded identity and an expiry. When the ten minutes are up, the file will still be there and will stop working on its own.

Mission 3: get into a node without SSH

You need to look at something on the filesystem of node-01 and you don't have (and don't want to set up) SSH access.

How do I debug a whole node, not just a Pod?

kubectl debug node/node-01 -it --image=ghcr.io/iximiuz/labs/nginx:alpine -- sh

Kubernetes creates a privileged Pod on that node with its filesystem mounted at /host. And since you're in, get something out of it: these are the three questions that get answered inside a node and can't be answered from outside.

df -h /host

How much disk it has left. It's the cause of the DiskPressure that evicts Pods from a node, and it shows up in no kubectl get.

ls /host/var/lib/kubelet/pods | wc -l

How many Pod directories the kubelet has left on disk. If that number never goes down, you have volumes that aren't being unmounted.

ls /host/etc/rancher/k3s

The configuration of k3s itself on that node: the files the kubelet reads at startup, which explain why the node behaves the way it does.

Leave with exit. The debug Pod (node-debugger-node-01-...) is left behind as evidence; in the real world you'd delete it when you're done.

Mission 4: waiting for something to really die

Scripts that delete resources tend to commit the same sin: assuming delete is instant, when it actually triggers a termination with a grace period. The correct wait, with no hand-written loops:

How do I wait for a resource to really disappear?

kubectl delete pod objetivo --wait=false
kubectl wait pod objetivo --for=delete --timeout=60s

The first command requests the deletion without blocking; the second doesn't return until the Pod is completely gone. It's the companion of the --for=condition=Available you used in the Declarative kubectl lesson, for the return trip.

Mission 5: the full schema, without leaving the terminal

Last mission, pure erudition. The documentation for every field of every resource lives inside the cluster itself:

How do I dump a resource's full schema in one go?

kubectl explain pod.spec --recursive | less

kubectl explain pod.spec.hostNetwork

The recursive form draws the whole tree of fields with their types; the targeted one explains one specific field. Exam question: what type is pod.spec.hostNetwork?

The other eight, for your notebook

How do I talk to the REST API directly, without kubectl's resource model?

kubectl get --raw /metrics | head

kubectl get --raw /api/v1/namespaces/tienda/pods

get --raw is the honest back door: any API server endpoint, raw. The first dumps the API server's own Prometheus metrics.

Did you know logs, exec and port-forward accept controllers directly?

kubectl logs deploy/coredns -n kube-system

You already used it in the Imperative kubectl lesson almost without noticing: deploy/, sts/ and rs/ work in all three commands, and kubectl picks a Pod for you.

How do I see the actual HTTP requests kubectl sends?

kubectl get pods --v=9

Verbosity 9: every request and response against the API, with URLs and bodies. The ultimate tool for understanding what kubectl does under the hood (and for copying those calls into your own scripts).

How do I follow an object's Events in a readable way?

kubectl events --for pod/<nombre> --watch

The modern replacement for the get events --field-selector you used in the Imperative kubectl lesson: same result, human syntax.

How do I change the tool kubectl diff uses?

To see it you need something to compare, so create an object and a manifest that differs from it in one field:

kubectl create configmap ajustes --from-literal=modo=lectura

cat << 'EOF' > manifiesto.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ajustes
data:
  modo: escritura
EOF

KUBECTL_EXTERNAL_DIFF="diff -u --color=always" kubectl diff -f manifiesto.yaml

One line in red (modo: lectura, what's there) and another in green (modo: escritura, what you propose); the two values mean "read" and "write". The variable accepts the command and its arguments, and what it receives is two directories: the current state and the proposed one. Any tool that compares two paths will do (colordiff, delta, vimdiff), as long as it's installed on the machine.

And a detail that surprises you in your first pipeline: kubectl diff exits with code 1 when it finds differences. It's not an error, it's its way of answering "yes, there are changes"; 0 means "this is already applied". A misplaced set -e turns that answer into an aborted rollout.

How do I read or modify only a resource's status?

kubectl get deployment <nombre> --subresource=status -o yaml

The status as an independent subresource: what the controllers write, separate from what you write.

How do I find out who owns each field of a resource?

kubectl get deployment <nombre> -o yaml --show-managed-fields

The managedFields that kubectl hides by default, and that everyone dismisses as noise, are actually Server-Side Apply's ownership record: which manager (kubectl, a controller, an operator) wrote each field. Key when two tools fight over a value.

How do I inject an arbitrary PodSpec into a temporary Pod, with no manifest?

kubectl run debug --image=ghcr.io/iximiuz/labs/nginx:alpine \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"node-01"}}}' -it --rm -- sh

--overrides merges raw JSON over the generated Pod. It'll ring a bell: it's the trick the ResourceQuotas and LimitRanges lesson used to build the greedy Pod the ResourceQuota rejected.

Summary

• Identity and access: create token (ephemeral, no Secrets) and config view --minify --flatten (a portable kubeconfig).
• Under the hood: get --raw, --v=9, --subresource=status, --show-managed-fields.
• Fine-grained operation: debug node/, wait --for=delete, events --for --watch, logs deploy/, run --overrides.
• And the humble gem: explain --recursive, the living documentation of your own cluster version.