Recently, I encountered a rather frustrating issue with my Ubuntu system running inside a Proxmox VE (PVE) virtual machine. The culprit? A seemingly innocent attempt to adjust the placement group (PG) count in my Ceph cluster. What started as a routine optimization turned into a full-blown system crash, leaving my Ubuntu installation unbootable and my disk unrecognizable. Here’s how I managed to recover it by restoring the partition table with gdisk and rebuilding GRUB.
Step 1: Disk Unrecognized – Enter gdisk
After some initial panic, I booted into an Ubuntu Live CD environment via PVE’s console to investigate. Running lsblk and fdisk -l revealed a chilling fact: the disk (/dev/sda) was visible, but its partition table was corrupt or missing. My LVM-based root filesystem, EFI partition, and everything else were gone—or so it seemed.
Thankfully, I remembered gdisk, a tool designed for managing GPT partition tables (which Ubuntu uses by default). I ran:
sudo gdisk /dev/sda
In the interactive menu, I chose the recovery option (r) and then attempted to rebuild the partition table from the backup GPT header (b). To my relief, gdisk detected the remnants of my original partitions: the EFI system partition (likely /dev/sda1), a possible /boot partition, and the LVM physical volume. After verifying and writing the restored table with w, I rebooted to see if the disk was recognized. Success—the partitions were back!
Step 2: The GRUB Dilemma
With the partition table restored, I could see my LVM volume group (ubuntu-vg) and logical volumes again. I mounted the root filesystem (/dev/ubuntu-vg/ubuntu-lv) and the EFI partition (/dev/sda1) to proceed with recovery. However, booting still failed—GRUB was missing or corrupted. This wasn’t surprising, as the crash might have interrupted a critical disk operation, leaving the bootloader in disarray.
To rebuild GRUB, I followed these steps in the Live environment:
1. Activate LVM and Mount Filesystems:Prerequisite:
Add an EFI disk for the VM.
Starting with a live CD, enter SHELL.
Mounting:
(1) The LVM disk
sudo vgscan
sudo vgchange -ay
sudo mount /dev/ubuntu-vg/ubuntu-lv /mnt
(2) The boot and efi folders
Identify the EFI file system: fdisk -l /dev/sda
sudo mount /dev/sda1 /mnt/boot/efi
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run
Formating:
(1) In this step you will need to check whether the kernel exsits or not with
ls -l /boot/vmlinuz-* /boot/initrd.img-*
(2) Then we use chroot /mnt to enter the file system.
Reinstall GRUB:
Since my VM was running an ARM64 architecture (confirmed with uname -m), I used:
sudo grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu
Initially, I hit a snag: “/boot/efi doesn’t look like an EFI partition.” This was because the EFI partition’s filesystem was damaged during the crash. I exited chroot, reformatted it with:
sudo mkfs.vfat -F32 /dev/sda1
Update GRUB Configuration:
update-grub
After this you can exit the chroot mode, and reboot the VM:
exit
sudo umount /mnt/dev /mnt/proc /mnt/sys /mnt/boot/efi /mnt
sudo reboot
Comment this blog if you have more questions.
Leave a Reply