Testbox Imaging (Backup / Restore)

Introduction

This document is explores deloying a very simple drive imaging solution to help avoid needing to manually reinstall testboxes when a disk goes bust or the OS install seems to be corrupted.

Definitions / Glossary

See AutomaticTestingRevamp.txt.

Objectives

  • Off site, no admin interaction (no need for ILOM or similar).
  • OS independent.
  • Space and bandwidth efficient.
  • As automatic as possible.
  • Logging.

Overview of the Solution

Here is a brief summary:

  • Always boot testboxes via PXE using PXELINUX.
  • Default configuration is local boot (hard disk / SSD)
  • Restore/backup action triggered by machine specific PXE config.
  • Boots special debian maintenance install off NFS.
  • A maintenance service (systemd style) does the work.
  • The service reads action from TFTP location and performs it.
  • When done the service removes the TFTP machine specific config and reboots the system.
Maintenance actions are:
  • backup
  • redo-backup
  • restore

Possible modifier that indicates a subset of disk on testboxes with other OSes installed. Support for partition level backup/restore is not explored here.

Storage Server

The storage server will have three areas used here. Using NFS for all three avoids extra work getting CIFS sharing right too (NFS is already a pain).

  1. /exports/testbox-tftp - TFTP config area. Read-write.
  2. /exports/testbox-backup - Images and logs. Read-write.
  3. /exports/testbox-nfsroot - Custom debian. Read-only, no root squash.

TFTP (/exports/testbox-tftp)

The testbox-tftp share needs to be writable, root squashing is okay.

We need files from both PXELINUX and SYSLINUX to make this work now. On a debian system, the pxelinux and syslinux packages needs to be installed. We actually do this further down when setting up the nfsroot, so it's possible to get them from there by postponing this step a little. On debian 8.6.0 the PXELINUX files are found in /usr/lib/PXELINUX and the SYSLINUX ones in /usr/lib/syslinux.

The initial PXE image as well as associated modules comes in three variants, BIOS, 32-bit EFI and 64-bit EFI. We'll only need the BIOS one for now. Perform the following copy operations:

cp /usr/lib/PXELINUX/pxelinux.0 testbox-tftp/
cp /usr/lib/syslinux/modules/*/ldlinux.* testbox-tftp/
cp /usr/lib/syslinux/modules/bios  testbox-tftp/
cp /usr/lib/syslinux/modules/efi32 testbox-tftp/
cp /usr/lib/syslinux/modules/efi64 testbox-tftp/

For simplicitly, all the testboxes boot using good old fashioned BIOS, no EFI. However, it doesn't really hurt to be prepared.

The PXELINUX related files goes in the root of the testbox-tftp share. (As mentioned further down, these can be installed on a debian system by running apt-get install pxelinux syslinux.) We need the *pxelinux.0 files typically found in /usr/lib/PXELINUX/ on debian systems (recent ones anyway). It is possible we may need one ore more fo the modules [1] that ships with PXELINUX/SYSLINUX, so do copy /usr/lib/syslinux/modules to testbox-tftp/modules as well.

The directory layout related to the configuration files is dictated by the PXELINUX configuration file searching algorithm [2]. Create a subdirectory pxelinux.cfg/ under testbox-tftp and create the world readable file default with the following content:

PATH bios
DEFAULT local-boot
LABEL local-boot
LOCALBOOT

This will make the default behavior to boot the local disk system.

Images and logs (/exports/testbox-backup)

The testbox-backup share needs to be writable, root squashing is okay.

We use a directory layout based on dotted decimal IP addresses here, so for a server with the IP 10.40.41.42 all its file will be under 10.40.41.42/.

Files found under 10.40.41.42/:

<hostname>
The name of the testbox (empty file). Help finding a testbox by name.
testbox-info.txt
Information about the testbox. Starting off with the name, decimal IP, PXELINUX style hexadecimal IP, and more.
maintenance.log
Maintenance log file recording what the maintenance service does.
disk-devices.lst
Optional list of disk devices to consider backuping up or restoring. This is intended for testboxes with additional disks that are used for other purposes and should touched.
sda.raw.gz
The gzipped raw copy of the sda device of the testbox.
sd[bcdefgh].raw.gz
The gzipped raw copy sdb, sdc, sde, sdf, sdg, sdh, etc if any of them exists and are disks/SSDs.
Note! If it turns out we can be certain to get a valid host name, we might just
switch to use the hostname as the directory name instead of the IP.

Debian NFS root (/exports/testbox-nfsroot)

The testbox-nfsroot share should be read-only and must not have root squashing enabled.

There are several ways of creating a debian nfsroot, but since we've got a tool like VirtualBox around we've just installed it in a VM, prepared it, and copied it onto the NFS server share.

As of writing debian 8.6.0 is current, so a minimal 64-bit install of it was done in a VM. After installation the following modifications was done:

  • apt-get install pxelinux syslinux initramfs-tools gddrescue joe and optionally apt-get install smbclient cifs-utils.

  • /etc/default/grub was modified to set GRUB_CMDLINE_LINUX_DEFAULT to "" instead of "quiet". This allows us to see messages during boot and perhaps spot why something doesn't work on a testbox. Regenerate the grub configuration file by running update-grub afterwards.

  • Create the directory /etc/systemd/system/getty@tty1.service.d and create the file noclear.conf in it with the following content:

    [Service]
TTYVTDisallocate=no

    This stops getty from clearing VT1 and let us see the tail of the boot up messages, which includes messages from the testbox-maintenance service.

  • Copy the testbox-maintenance.sh file found in the same directory as this document to /root/scripts/ (need to create the dir) and make it executable.

  • Create the systemd service file for the maintenance service as /etc/systemd/system/textbox-maintenance.service with the content:

    [Unit]
Description=Testbox Maintenance
After=network.target
Before=getty@tty1.service

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=/root/scripts/testbox-maintenance.sh
ExecStartPre=/bin/echo -e \033%G
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/tmp
Environment=TERM=xterm
StandardOutput=journal+console

[Install]
WantedBy=multi-user.target

  • Mount the testbox-nfsroot under /mnt/ with write privileges. (The write privileges are temporary - don't forget to remove them later on.):

    mount -t nfs myserver.com:/exports/testbox-nfsroot

  • Copy the debian root and dev file system onto nfsroot. If you have ssh access to the NFS server, the quickest way to do it is to use tar:

    tar -cz --one-file-system -f /mnt/testbox-maintenance-nfsroot.tar.gz . dev/

    An alternative is cp -ax . /mnt/. &&  cp -ax dev/. /mnt/dev/. but this is quite a bit slower, obviously.

  • chroot into the nfsroot: chroot /mnt/

    • Enable our service: systemctl enable testbox-maintenance.service

    • Edit /etc/initramfs-tools/initramfs.conf and change the MODULES value from most to netboot.

    • Append aufs to /etc/initramfs-tools/modules. The advanced multi-layered unification filesystem (aufs) enables us to use a read-only NFS root. [3] [4] [5]

    • Create /etc/initramfs-tools/scripts/init-bottom/00_aufs_init as an executable file with the following content:

      # Don't run during update-initramfs:
case "$1" in
    prereqs)
        exit 0;
        ;;
esac

modprobe aufs
mkdir -p /ro /rw /aufs
mount -t tmpfs tmpfs /rw -o noatime,mode=0755
mount --move $rootmnt /ro
mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro
mkdir -p /aufs/rw /aufs/ro
mount --move /ro /aufs/ro
mount --move /rw /aufs/rw
mount --move /aufs /root
exit 0

    • Update the init ramdisk: update-initramfs -u -k all

    • Either copy the initrd.img and corresponding kernel to the testbox-tftp/maintenance-boot/ directory, or create testbox-tftp/maintenance-boot as a symbolic link to testbox-nfsroot/boot/.

    • Recreate /etc/fstab with:

      proc          /proc    proc    defaults        0 0
/dev/nfs      /        nfs     defaults        1 1
# Mount tftp and backup too.

    • xxx

  • Testing the setup from a VM is kind of useful:

    • Create a VM using the 64-bit debian profile. Let's call it "pxe-vm".

    • Mount the TFTP share somewhere, like M: or /mnt/testbox-tftp.

    • Reconfigure the NAT DHCP and TFTP bits:

      VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/AboveDriver       NAT
VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Action            mergeconfig
VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/TFTPPrefix M:/
VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/BootFile   pxelinux.0

    • Create the file testbox-tftp/pxelinux.cfg/0A00020F containing:

      PATH bios
DEFAULT maintenance
LABEL maintenance
  MENU LABEL Maintenance (NFS)
  KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
  APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 ro ip=dhcp aufs=tmpfs \
         boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/exports/testbox-nfsroot
LABEL local-boot
LOCALBOOT
[1]See http://www.syslinux.org/wiki/index.php?title=Category:Modules
[2]See http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration
[3]See https://en.wikipedia.org/wiki/Aufs
[4]See http://shitwefoundout.com/wiki/Diskless_ubuntu
[5]See http://debianaddict.com/2012/06/19/diskless-debian-linux-booting-via-dhcppxenfstftp/
Status:$Id: TestBoxImaging.html 64498 2016-11-01 04:39:14Z vboxsync $
Copyright:Copyright (C) 2010-2016 Oracle Corporation.