Challenge, Medium,  on  Kubernetes

Provision Dynamic NFS CSI Storage to Restore Stuck Deployments

Omkar Shelke
by  Omkar Shelke

Scenario

The audit-trail-writer deployment in the audit namespace runs 3 replicas where each pod continuously writes structured audit logs to a shared NFS volume on nfs-storage-server, capturing its own pod name, the node it is scheduled on, and its pod IP.

All pods are stuck in Pending because the deployment references a PersistentVolumeClaim named audit-trail-pvc that does not exist, and no StorageClass has been created to provision it.

The NFS server is already running on nfs-storage-server (172.16.0.5) at export path /srv/nfs/k8s, and the NFS CSI driver (nfs.csi.k8s.io) is already installed in the cluster.

Create the missing StorageClass and PersistentVolumeClaim to unblock the deployment and get all 3 pods writing audit logs to shared storage.

Task

1. Create a StorageClass named platform-nfs with the following specification:

  • Provisioner: nfs.csi.k8s.io
  • Parameters:
    • server: 172.16.0.5
    • share: /srv/nfs/k8s
  • Reclaim policy: Delete
  • Volume binding mode: Immediate
  • Allow volume expansion: true

2. Create a PersistentVolumeClaim named audit-trail-pvc in the audit namespace with the following specification:

  • StorageClass: platform-nfs
  • Access mode: ReadWriteMany
  • Storage request: 1Gi
Kubernetes pods writing audit logs to shared NFS storage using PVC and StorageClass

The audit-trail-writer pods mount the shared NFS-backed PVC (audit-trail-pvc) provisioned by the platform-nfs StorageClass and write logs to shared storage.

Once the PVC is bound, the stalled pods will automatically recover and start writing audit logs to the shared NFS volume on nfs-storage-server.

⏳ Wait up to 1 minutes for the environment to fully initialise before starting — NFS server setup and CSI driver installation run in the background.

Hint 1 — StorageClass

Create a StorageClass using the nfs.csi.k8s.io provisioner pointing to the NFS server on dev-machine:

parameters:
  server: 172.16.0.5
  share: /srv/nfs/k8s
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
Hint 2 — PersistentVolumeClaim

Use ReadWriteMany access mode and request 1Gi of storage — all 3 pods run on different nodes and must mount the same NFS volume simultaneously.

resources:
  requests:
    storage: 1Gi

Test Cases

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge provision-dynamic-nfs-storage-to-restore-stuck-deployments-f53f0211

This will create a local copy of the challenge content in a directory named provision-dynamic-nfs-storage-to-restore-stuck-deployments-f53f0211. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge provision-dynamic-nfs-storage-to-restore-stuck-deployments-f53f0211

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge