Challenge, Medium,  on  Kubernetes

Kubernetes - PV Reclaim Policies and Recovery

Kaloyan Preslavski
by  Kaloyan Preslavski

Over its lifetime, a PV passes through several phases:

  • Available - no PVC has claimed it
  • Bound - a PVC has claimed it
  • Released - the PVC that claimed it was deleted, but the PV still exists

Every PV has a reclaim policy that determines what happens when its PVC is deleted. The Delete policy removes the PV and its underlying storage, so the PV never reaches Released. The Retain policy keeps the PV and its data in place. The PV moves to Released and an administrator decides what happens to the data next.

This challenge demonstrates both policies using local-path, the built-in provisioner in k3s, and covers two recovery patterns for a Retained PV.

Task 1 - Provision a Volume

The local-path provisioner creates a PV automatically once a pod that uses the PVC gets scheduled to a node. Create a PVC and a pod that mounts it, then confirm the provisioner created a PV.

Steps:

  • Create a PVC named pvc-delete without a storageClassName field, capacity 1Gi, access mode ReadWriteOnce
  • Create a pod named pod-delete (image busybox, command sleep 3600) mounting pvc-delete at /data
  • Confirm the PVC is Bound and note the PV name

Task 2 - Delete Reclaim Policy

The default reclaim policy for local-path is Delete. When a PVC is deleted, the provisioner deletes the PV and the underlying storage directory.

Steps:

  • Delete pod-delete, then pvc-delete
  • Verify the PV is also gone

Task 3 - Retain Reclaim Policy

Create a StorageClass with reclaimPolicy: Retain so the PV survives PVC deletion.

Steps:

  • Create a StorageClass named sc-retain with provisioner: rancher.io/local-path, volumeBindingMode: WaitForFirstConsumer, and reclaimPolicy: Retain
  • Create a PVC named pvc-retain with storageClassName: sc-retain, capacity 1Gi, access mode ReadWriteOnce
  • Create a pod named retain-pod (image busybox, command sleep 3600) mounting pvc-retain at /data
  • Wait for the pod to be Running
  • Write important-data to /data/recover.txt inside the pod (Task 4 recovers it)
  • Delete the pod first, then the PVC
  • Verify the PV is still around in Released state

Task 4 - Recover the Released PV

After Task 3, the PV is in Released state. The claimRef field still references the deleted PVC and blocks binding to a new claim. Removing claimRef transitions the PV to Available. The new PVC does not name the PV directly. It just requests storageClassName: sc-retain, and Kubernetes matches it to the only Available PV of that class by capacity and access mode. The data on disk is untouched.

Steps:

  • Run kubectl get pv and find the PV in Released state
  • Run kubectl edit pv <name> and delete the claimRef block from spec, then save
  • Confirm the PV transitions to Available
  • Create a PVC named pvc-recover with storageClassName: sc-retain, capacity 1Gi, access mode ReadWriteOnce
  • Create a pod named recover-pod (image busybox, command sleep 3600) mounting pvc-recover at /data
  • Read /data/recover.txt from inside the pod and confirm the data is still there

Task 5 - Recover by Creating a New PV

Task 4 patched claimRef on the existing PV object. This time, leave the Released PV alone and create a new PV pointing to the same storage directory. Reuse the sc-retain StorageClass from Task 3, so the new PV keeps the Retain reclaim policy too.

Pin pvc-recover-2 to pv-new directly instead of leaving Kubernetes to match them by class, capacity, and access mode. You already know the exact PV you want, so name it.

Steps:

  • Create a PVC named pvc-retain-2 with storageClassName: sc-retain, capacity 1Gi, access mode ReadWriteOnce
  • Create a pod named pod-retain-2 (image busybox, command sleep 3600) mounting pvc-retain-2 at /data
  • Write important-data to /data/recover.txt inside the pod
  • Delete pod-retain-2, then pvc-retain-2
  • Find the Released PV and get its spec.hostPath.path
  • Create a PV named pv-new with storageClassName: sc-retain, hostPath.path set to that directory, capacity 1Gi, access mode ReadWriteOnce
  • Create a PVC named pvc-recover-2 with storageClassName: sc-retain, volumeName set to pv-new, capacity 1Gi, access mode ReadWriteOnce
  • Create a pod named recover-pod-2 (image busybox, command sleep 3600) mounting pvc-recover-2 at /data
  • Read /data/recover.txt from inside recover-pod-2 and confirm the data is still there

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge kubernetes-pv-reclaim-policies-5395f28c

This will create a local copy of the challenge content in a directory named kubernetes-pv-reclaim-policies-5395f28c. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge kubernetes-pv-reclaim-policies-5395f28c

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge