Reclaim a Retained PV with a New PVC and Restore a Lost MariaDB Database

Scenario

A cleanup script accidentally deleted the prod-vault MariaDB deployment and its PersistentVolumeClaim (prod-data-pvc) in the prod-data namespace.

The PersistentVolume (prod-data-pv) was configured with a Retain reclaim policy — so the volume and the database files survived the deletion.

The PV has a nodeAffinity pinned to node-02.

The database files are still present on that node:

node-02
└── /mnt/prod-mariadb
    ├── aria_log.00000001
    ├── aria_log_control
    ├── ddl_recovery.log
    ├── ib_buffer_pool
    ├── ib_logfile0
    ├── ibdata1
    ├── ibtmp1
    ├── multi-master.info
    ├── mysql/
    ├── mysql_upgrade_info
    ├── performance_schema/
    ├── proddb/
    └── sys/

Your job is to recover the application without creating a new PV and without losing any data.

Task

Fix the PV so it is ready to accept a new PVC.
Create a new PVC named prod-data-pvc in the prod-data namespace that binds to the existing prod-data-pv:
Field Value
Name prod-data-pvc
Namespace prod-data
Access Mode ReadWriteOnce
Storage Request 5Gi
Storage Class "" (empty — disables dynamic provisioning)
Volume Name prod-data-pv
Recreate the application using the manifest located at /home/laborant/mariadb.yaml.

Field	Value
Name	`prod-data-pvc`
Namespace	`prod-data`
Access Mode	`ReadWriteOnce`
Storage Request	`5Gi`
Storage Class	`""` (empty — disables dynamic provisioning)
Volume Name	`prod-data-pv`

Verify the inventory data survived by running:

POD=$(kubectl get pods -n prod-data -l app=prod-vault -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it "$POD" -n prod-data -- mysql -uroot -pprod-root-pass -e "USE proddb; SELECT * FROM inventory;"

All 30 inventory records must be present.

+----+-----------------+-------------+----------+----------+
| id | item_name       | category    | quantity | price    |
+----+-----------------+-------------+----------+----------+
|  1 | Laptop          | Electronics |       10 | 65000.00 |
|  2 | Mouse           | Electronics |       50 |   799.00 |
|  3 | Keyboard        | Electronics |       30 |  1499.00 |
|  4 | Monitor         | Electronics |       15 | 12500.00 |
|  5 | Office Chair    | Furniture   |       20 |  5500.00 |
| ... (25 additional rows omitted) ...                     |
| 30 | Electric Kettle | Kitchen     |       13 |  1899.00 |
+----+-----------------+-------------+----------+----------+

Hint 1 — Check the Current PV State

Start by inspecting what happened to the PV after the PVC was deleted:

kubectl get pv prod-data-pv
kubectl describe pv prod-data-pv

Look at the Status and Claim fields in the output. A PV with Retain policy does not go back to Available on its own — understanding why is the key to this recovery.

Documentation

Hint 2 — Get the PV Ready for Binding

Dump the full PV spec and look carefully at what's inside:

kubectl get pv prod-data-pv -o yaml

The PV is currently stuck in Released state. A new PVC can only bind to a PV that is in Available state — so you need to get it there first.

Look closely at the spec and find what is still referencing the old deleted PVC. You'll need to clear it to move the PV from Released → Available.

You can modify a live resource without recreating it using either:

kubectl edit — opens the full manifest in your editor for manual changes
kubectl patch — applies a targeted change directly from the command line

Once done, confirm the status before moving on:

kubectl get pv prod-data-pv

Documentation

Hint 3 — Understand What Controls PVC Binding

Check what fields are available on a PVC spec:

kubectl explain pvc.spec

Two fields matter here: one that targets a specific volume by name, and one that controls whether Kubernetes uses dynamic provisioning. Setting them correctly ensures your new PVC binds to prod-data-pv and nothing else.

Here is the structure to follow when writing your PVC spec:

spec:
  accessModes:
    - ReadWriteOnce          # must match the PV's access mode
  resources:
    requests:
      storage: 5Gi           # must match the PV's capacity
  volumeName: <pv-name>      # pin to a specific PV by name
  storageClassName: ""       # empty string disables dynamic provisioning

Documentation

Hint 4 — Verify the Data Survived

Once the deployment is running, get the pod name and confirm the database records are intact:

POD=$(kubectl get pod -n prod-data -l app=prod-vault \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it $POD -n prod-data -- \
  mysql -uroot -pprod-root-pass -e "USE proddb; SELECT COUNT(*) FROM inventory;"

The query must return 30. If it does, the PV recovery was successful and no data was lost.

You can also view all records to confirm:

kubectl exec -it $POD -n prod-data -- \
  mysql -uroot -pprod-root-pass -e "USE proddb; SELECT * FROM inventory;"

Documentation

Reclaim a Retained PV with a New PVC and Restore a Lost MariaDB Database

Scenario

Task

⚒ Test Cases