Create a Custom Resource Definition and Verify It in the Kubernetes API
Scenario
Your team runs an internal platform that keeps website configurations in Kubernetes. The platform team wants a Website resource type, so that operators can create and list Website objects with the same tools they use for built-in resources such as Deployments.
An audit job will read these objects straight from the Kubernetes API, with the token of the default service account in the default namespace. Before the job goes live, the team wants proof that the new endpoint works for this account: the raw API response, saved to a file.
Work on dev-machine. Its kubectl is already configured with cluster admin access.
Task
Register the Website resource type with a CustomResourceDefinition (CRD):
| Setting | Requirement |
|---|---|
| Group | apps.acme.com |
| Version | v1, served and stored |
| Kind | Website |
| Plural name | websites |
| Singular name | website |
| Short name | ws |
| Scope | Namespaced |
The v1 schema must accept two fields under spec: url, a required string, and replicas, an optional integer.
Give the default service account in the default namespace permission to get and list Website objects in all namespaces. Put the permission in a ClusterRole named website-reader, and grant it with a ClusterRoleBinding named website-reader-binding.
Finally, prove that the endpoint works. Use curl, authenticated as the default service account, to request the API endpoint that lists Website objects in all namespaces. Save the JSON response body to /home/laborant/crd-live.json on dev-machine.
Optional: The API server on cplane-01 records the requests for websites in the audit log /var/log/k3s-audit.log, which you can read with sudo. After the checks pass, find your curl request in it and look at the user, the client and the ClusterRoleBinding that the API server recorded. You can skip this part without affecting the test cases.
Hint 1: Writing the CRD
A CRD is a Kubernetes object like any other, so you write it as a manifest and apply it. kubectl explain describes every field:
kubectl explain customresourcedefinition.spec
kubectl explain customresourcedefinition.spec.names
kubectl explain customresourcedefinition.spec.versions.schema
The overall shape:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: <plural>.<group>
spec:
group: <group>
scope: <scope>
names:
# plural, singular, kind and short names
versions:
- name: <version>
served: true
storage: true
schema:
openAPIV3Schema:
# an object whose spec property holds the fields
metadata.name must be the plural name, a dot and the group. kubectl apply rejects the CRD otherwise. In an OpenAPI schema, the required list belongs to the object that holds the fields.
Docs:
- Extend the Kubernetes API with CustomResourceDefinitions (Kubernetes)
- Custom Resources (Kubernetes)
Hint 2: Checking the CRD
The API server serves the new endpoint once the CRD has the condition Established set to True. The conditions are at the end of the describe output:
kubectl describe crd <name>
NamesAccepted is False when another CRD already uses one of the names. When the CRD is ready, the new resource type is also listed among the API resources of its group:
kubectl api-resources --api-group=<group>
Docs:
- Create a CustomResourceDefinition (Kubernetes)
Hint 3: RBAC for a custom resource
RBAC rules for a custom resource use the group and the plural name from the CRD, the same way as for built-in resources. Both objects can be created with kubectl create, and the help pages show examples:
kubectl create clusterrole --help
kubectl create clusterrolebinding --help
A service account is named <namespace>:<name> in kubectl create clusterrolebinding. Check the result as that account:
kubectl auth can-i <verb> <plural>.<group> --as=system:serviceaccount:<namespace>:<name>
Docs:
- Using RBAC Authorization (Kubernetes)
- Referring to resources (Kubernetes)
Hint 4: Calling the API with curl
The API server does not run on dev-machine. kubectl already knows its address:
kubectl cluster-info
The path of a list request across all namespaces is /apis/<group>/<version>/<plural>.
The API server accepts a service account token in the Authorization: Bearer <token> header. kubectl create token prints a short-lived token for a service account. The shortest duration it accepts is 10 minutes.
Useful curl flags:
-Hadds a request header.-kskips the check of the server certificate. It is signed by the cluster's own CA, whichcurldoes not trust by default.-o <file>saves the response body to a file.
Docs:
- Access the Kubernetes API without kubectl proxy (Kubernetes)
- kubectl create token (Kubernetes)
Hint 5: When the request fails
Run the request once without -o, so that the response body is printed. On an error, the API server returns a Status object, and its code and message explain the problem:
401: the request has no valid token.403: the token is valid, but RBAC does not allow the request. The message names the user, the verb and the resource. Compare them with your ClusterRole and ClusterRoleBinding. A typo in the path also ends here, because the API server checks the permission before it looks for the resource.