Recover an Ingestion Service After a VM Upgrade
The ingest-01 server runs ingestd, a service that processes events from an upstream queue.
The incoming data amounts to roughly 3 GiB per day,
and the service keeps a local disk cache of the last ten days of events for fast lookups and duplicate detection.
The cache can be rebuilt from the primary data store if needed.
As the workload grew, the team decided to give ingest-01 more CPU and memory.
Earlier today, they moved the server from an older VM family to a newer one, keeping its existing disks.
The server booted successfully, and ingestd initially appeared to be working, but stopped shortly afterward.
Find out what went wrong and restore the server to a healthy state.
Hint 1
Check the service's current state with systemctl status ingest and its recent logs with sudo journalctl -u ingest.
What caused the service to stop?
Hint 2
Use df -h to inspect the available filesystems and their free space.
The service stores its cache under /var/lib/ingest.
Given the incoming data rate and the cache retention period, how much disk space does it need?
Does the filesystem currently holding the cache have enough room?
Hint 3
Use lsblk -f to list the server's block devices and filesystems, including those that aren't mounted.
Compare the available filesystems with the entries in /etc/fstab.
Is every filesystem mounted where it should be?
Hint 4
Check the current boot's logs with sudo journalctl -b, paying attention to failed mounts.
Block device names such as /dev/xvdf or /dev/nvme1n1 depend on the storage driver.
Moving to a different VM family may change these names even when the underlying disks stay the same.
Does /etc/fstab still refer to the correct devices?
Hint 5
If the service is working again but the root filesystem is still unusually full, compare the output of df and du.
For example:
df -h /
sudo du -xsh /
If the filesystem reports considerably more used space than du can account for, where could the missing gigabytes be?
Hint 6
Mounting a filesystem on an existing directory hides whatever was already in that directory. The hidden files aren't deleted, and they still occupy disk space.
A non-recursive bind mount of / lets you inspect the underlying root filesystem without unmounting the data disk:
sudo mkdir -p /mnt/root-view
sudo mount --bind / /mnt/root-view
Check what is stored under /mnt/root-view/var/lib/ingest.
Be careful to remove only the misplaced cache files from the root filesystem, not the existing cache on the mounted data disk.
Unmount the temporary view when you're done.
Finally, make sure the fix is permanent. When you're confident in the configuration, reboot the server:
sudo reboot
Hint 7
Device paths assigned by the kernel can change between VM types and even between boots.
A filesystem's UUID or label provides a more stable reference.
Use lsblk -f or sudo blkid to find it, and update /etc/fstab to use UUID=... instead of a device path.
Hint 8
Run sudo findmnt --verify to check /etc/fstab for configuration problems.
Also confirm that the correct filesystem is actually mounted at /var/lib/ingest.
Keep the nofail option so the server can still boot if the data disk is unavailable.
This helps preserve remote access to the server, although it doesn't prevent the ingestion service itself from starting without its data filesystem.
