| 1 |
|
|---|
| 2 | Testbox Imaging (Backup / Restore)
|
|---|
| 3 | ==================================
|
|---|
| 4 |
|
|---|
| 5 |
|
|---|
| 6 | Introduction
|
|---|
| 7 | ------------
|
|---|
| 8 |
|
|---|
| 9 | This document is explores deloying a very simple drive imaging solution to help
|
|---|
| 10 | avoid needing to manually reinstall testboxes when a disk goes bust or the OS
|
|---|
| 11 | install seems to be corrupted.
|
|---|
| 12 |
|
|---|
| 13 |
|
|---|
| 14 | Definitions / Glossary
|
|---|
| 15 | ======================
|
|---|
| 16 |
|
|---|
| 17 | See AutomaticTestingRevamp.txt.
|
|---|
| 18 |
|
|---|
| 19 |
|
|---|
| 20 | Objectives
|
|---|
| 21 | ==========
|
|---|
| 22 |
|
|---|
| 23 | - Off site, no admin interaction (no need for ILOM or similar).
|
|---|
| 24 | - OS independent.
|
|---|
| 25 | - Space and bandwidth efficient.
|
|---|
| 26 | - As automatic as possible.
|
|---|
| 27 | - Logging.
|
|---|
| 28 |
|
|---|
| 29 |
|
|---|
| 30 | Overview of the Solution
|
|---|
| 31 | ========================
|
|---|
| 32 |
|
|---|
| 33 | Here is a brief summary:
|
|---|
| 34 |
|
|---|
| 35 | - Always boot testboxes via PXE using PXELINUX.
|
|---|
| 36 | - Default configuration is local boot (hard disk / SSD)
|
|---|
| 37 | - Restore/backup action triggered by machine specific PXE config.
|
|---|
| 38 | - Boots special debian maintenance install off NFS.
|
|---|
| 39 | - A maintenance service (systemd style) does the work.
|
|---|
| 40 | - The service reads action from TFTP location and performs it.
|
|---|
| 41 | - When done the service removes the TFTP machine specific config
|
|---|
| 42 | and reboots the system.
|
|---|
| 43 |
|
|---|
| 44 | Maintenance actions are:
|
|---|
| 45 | - backup
|
|---|
| 46 | - backup-again
|
|---|
| 47 | - restore
|
|---|
| 48 | - refresh-info
|
|---|
| 49 | - rescue
|
|---|
| 50 |
|
|---|
| 51 | Possible modifier that indicates a subset of disk on testboxes with other OSes
|
|---|
| 52 | installed. Support for partition level backup/restore is not explored here.
|
|---|
| 53 |
|
|---|
| 54 |
|
|---|
| 55 | How to use
|
|---|
| 56 | ----------
|
|---|
| 57 |
|
|---|
| 58 | To perform one of the above maintenance actions on a testbox copy the
|
|---|
| 59 |
|
|---|
| 60 |
|
|---|
| 61 | Storage Server
|
|---|
| 62 | ==============
|
|---|
| 63 |
|
|---|
| 64 | The storage server will have three areas used here. Using NFS for all three
|
|---|
| 65 | avoids extra work getting CIFS sharing right too (NFS is already a pain).
|
|---|
| 66 |
|
|---|
| 67 | 1. /export/testbox-tftp - TFTP config area. Read-write.
|
|---|
| 68 | 2. /export/testbox-backup - Images and logs. Read-write.
|
|---|
| 69 | 3. /export/testbox-nfsroot - Custom debian. Read-only, no root squash.
|
|---|
| 70 |
|
|---|
| 71 |
|
|---|
| 72 | TFTP (/export/testbox-tftp)
|
|---|
| 73 | ============================
|
|---|
| 74 |
|
|---|
| 75 | The testbox-tftp share needs to be writable, root squashing is okay.
|
|---|
| 76 |
|
|---|
| 77 | We need files from both PXELINUX and SYSLINUX to make this work now. On a
|
|---|
| 78 | debian system, the ``pxelinux`` and ``syslinux`` packages needs to be
|
|---|
| 79 | installed. We actually do this further down when setting up the nfsroot, so
|
|---|
| 80 | it's possible to get them from there by postponing this step a little. On
|
|---|
| 81 | debian 8.6.0 the PXELINUX files are found in ``/usr/lib/PXELINUX`` and the
|
|---|
| 82 | SYSLINUX ones in ``/usr/lib/syslinux``.
|
|---|
| 83 |
|
|---|
| 84 | The initial PXE image as well as associated modules comes in three variants,
|
|---|
| 85 | BIOS, 32-bit EFI and 64-bit EFI. We'll only need the BIOS one for now.
|
|---|
| 86 | Perform the following copy operations::
|
|---|
| 87 |
|
|---|
| 88 | cp /usr/lib/PXELINUX/pxelinux.0 /mnt/testbox-tftp/
|
|---|
| 89 | cp /usr/lib/syslinux/modules/*/ldlinux.* /mnt/testbox-tftp/
|
|---|
| 90 | cp -R /usr/lib/syslinux/modules/bios /mnt/testbox-tftp/
|
|---|
| 91 | cp -R /usr/lib/syslinux/modules/efi32 /mnt/testbox-tftp/
|
|---|
| 92 | cp -R /usr/lib/syslinux/modules/efi64 /mnt/testbox-tftp/
|
|---|
| 93 |
|
|---|
| 94 |
|
|---|
| 95 | For simplicitly, all the testboxes boot using good old fashioned BIOS, no EFI.
|
|---|
| 96 | However, it doesn't really hurt to be prepared.
|
|---|
| 97 |
|
|---|
| 98 | The PXELINUX related files goes in the root of the testbox-tftp share. (As
|
|---|
| 99 | mentioned further down, these can be installed on a debian system by running
|
|---|
| 100 | ``apt-get install pxelinux syslinux``.) We need the ``*pxelinux.0`` files
|
|---|
| 101 | typically found in ``/usr/lib/PXELINUX/`` on debian systems (recent ones
|
|---|
| 102 | anyway). It is possible we may need one ore more fo the modules [1]_ that
|
|---|
| 103 | ships with PXELINUX/SYSLINUX, so do copy ``/usr/lib/syslinux/modules`` to
|
|---|
| 104 | ``testbox-tftp/modules`` as well.
|
|---|
| 105 |
|
|---|
| 106 |
|
|---|
| 107 | The directory layout related to the configuration files is dictated by the
|
|---|
| 108 | PXELINUX configuration file searching algorithm [2]_. Create a subdirectory
|
|---|
| 109 | ``pxelinux.cfg/`` under ``testbox-tftp`` and create the world readable file
|
|---|
| 110 | ``default`` with the following content::
|
|---|
| 111 |
|
|---|
| 112 | PATH bios
|
|---|
| 113 | DEFAULT local-boot
|
|---|
| 114 | LABEL local-boot
|
|---|
| 115 | LOCALBOOT
|
|---|
| 116 |
|
|---|
| 117 | This will make the default behavior to boot the local disk system.
|
|---|
| 118 |
|
|---|
| 119 | Create ``pxelinux.cfg/do-backup``, ``pxelinux.cfg/do-backup-again``,
|
|---|
| 120 | ``pxelinux.cfg/do-restore``, ``pxelinux.cfg/do-refresh-info``, and
|
|---|
| 121 | ``pxelinux.cfg/do-rescue`` configuration files on the form::
|
|---|
| 122 |
|
|---|
| 123 | PATH bios
|
|---|
| 124 | DEFAULT maintenance
|
|---|
| 125 | LABEL maintenance
|
|---|
| 126 | MENU LABEL Maintenance (NFS)
|
|---|
| 127 | KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
|
|---|
| 128 | APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 testbox-action-backup ro ip=dhcp aufs=tmpfs boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/export/testbox-nfsroot,ro nfsvers=3 nfsrootdebug
|
|---|
| 129 | LABEL local-boot
|
|---|
| 130 | LOCALBOOT
|
|---|
| 131 |
|
|---|
| 132 | When you want to preform an action on a testbox, copy the ``do-<action>`` to
|
|---|
| 133 | ``pxeclient.cfg/<HEX-ip-addr>`` and trigger a boot of the testbox. The machine
|
|---|
| 134 | config will be removed automatically once the action has been successfully
|
|---|
| 135 | completed.
|
|---|
| 136 |
|
|---|
| 137 |
|
|---|
| 138 |
|
|---|
| 139 | Images and logs (/export/testbox-backup)
|
|---|
| 140 | =========================================
|
|---|
| 141 |
|
|---|
| 142 | The testbox-backup share needs to be writable, root squashing is okay.
|
|---|
| 143 |
|
|---|
| 144 | In the root there must be a file ``testbox-backup`` so we can easily tell
|
|---|
| 145 | whether we've actually mounted the share or are just staring at an empty mount
|
|---|
| 146 | point directory.
|
|---|
| 147 |
|
|---|
| 148 | The ``testbox-maintenance.sh`` script maintains a global log in the root
|
|---|
| 149 | directory that's called ``maintenance.log``. Errors will be logged there as
|
|---|
| 150 | well as a ping and the action.
|
|---|
| 151 |
|
|---|
| 152 | We use a directory layout based on dotted decimal IP addresses here, so for a
|
|---|
| 153 | server with the IP 10.40.41.42 all its file will be under ``10.40.41.42/``:
|
|---|
| 154 |
|
|---|
| 155 | ``<hostname>``
|
|---|
| 156 | The name of the testbox (empty file). Help finding a testbox by name.
|
|---|
| 157 |
|
|---|
| 158 | ``testbox-info.txt``
|
|---|
| 159 | Information about the testbox. Starting off with the name, decimal IP,
|
|---|
| 160 | PXELINUX style hexadecimal IP, and more.
|
|---|
| 161 |
|
|---|
| 162 | ``maintenance.log``
|
|---|
| 163 | Maintenance log file recording what the maintenance service does.
|
|---|
| 164 |
|
|---|
| 165 | ``disk-devices.lst``
|
|---|
| 166 | Optional list of disk devices to consider backuping up or restoring. This is
|
|---|
| 167 | intended for testboxes with additional disks that are used for other purposes
|
|---|
| 168 | and should touched.
|
|---|
| 169 |
|
|---|
| 170 | ``sda.raw.gz``
|
|---|
| 171 | The gzipped raw copy of the sda device of the testbox.
|
|---|
| 172 |
|
|---|
| 173 | ``sd[bcdefgh].raw.gz``
|
|---|
| 174 | The gzipped raw copy sdb, sdc, sde, sdf, sdg, sdh, etc if any of them exists
|
|---|
| 175 | and are disks/SSDs.
|
|---|
| 176 |
|
|---|
| 177 |
|
|---|
| 178 | Note! If it turns out we can be certain to get a valid host name, we might just
|
|---|
| 179 | switch to use the hostname as the directory name instead of the IP.
|
|---|
| 180 |
|
|---|
| 181 |
|
|---|
| 182 | Debian NFS root (/export/testbox-nfsroot)
|
|---|
| 183 | ==========================================
|
|---|
| 184 |
|
|---|
| 185 | The testbox-nfsroot share should be read-only and must **not** have root
|
|---|
| 186 | squashing enabled.
|
|---|
| 187 |
|
|---|
| 188 | There are several ways of creating a debian nfsroot, but since we've got a
|
|---|
| 189 | tool like VirtualBox around we've just installed it in a VM, prepared it,
|
|---|
| 190 | and copied it onto the NFS server share.
|
|---|
| 191 |
|
|---|
| 192 | As of writing debian 8.6.0 is current, so a minimal 64-bit install of it was
|
|---|
| 193 | done in a VM. After installation the following modifications was done:
|
|---|
| 194 |
|
|---|
| 195 | - ``apt-get install pxelinux syslinux initramfs-tools zip gddrescue joe``
|
|---|
| 196 | and optionally ``apt-get install smbclient cifs-utils``.
|
|---|
| 197 |
|
|---|
| 198 |
|
|---|
| 199 | - ``/etc/default/grub`` was modified to set ``GRUB_CMDLINE_LINUX_DEFAULT`` to
|
|---|
| 200 | ``""`` instead of ``"quiet"``. This allows us to see messages during boot
|
|---|
| 201 | and perhaps spot why something doesn't work on a testbox. Regenerate the
|
|---|
| 202 | grub configuration file by running ``update-grub`` afterwards.
|
|---|
| 203 |
|
|---|
| 204 | - Create the directory ``/etc/systemd/system/[email protected]`` and create
|
|---|
| 205 | the file ``noclear.conf`` in it with the following content::
|
|---|
| 206 |
|
|---|
| 207 | [Service]
|
|---|
| 208 | TTYVTDisallocate=no
|
|---|
| 209 |
|
|---|
| 210 | This stops getty from clearing VT1 and let us see the tail of the boot up
|
|---|
| 211 | messages, which includes messages from the testbox-maintenance service.
|
|---|
| 212 |
|
|---|
| 213 | - Mount the testbox-nfsroot under ``/mnt/`` with write privileges. (The write
|
|---|
| 214 | privileges are temporary - don't forget to remove them later on.)::
|
|---|
| 215 |
|
|---|
| 216 | mount -t nfs myserver.com:/export/testbox-nfsroot
|
|---|
| 217 |
|
|---|
| 218 | Note! Adding ``-o nfsvers=3`` may help with some NTFv4 servers.
|
|---|
| 219 |
|
|---|
| 220 | - Copy the debian root and dev file system onto nfsroot. If you have ssh
|
|---|
| 221 | access to the NFS server, the quickest way to do it is to use ``tar``::
|
|---|
| 222 |
|
|---|
| 223 | tar -cz --one-file-system -f /mnt/testbox-maintenance-nfsroot.tar.gz . dev/
|
|---|
| 224 |
|
|---|
| 225 | An alternative is ``cp -ax . /mnt/. && cp -ax dev/. /mnt/dev/.`` but this
|
|---|
| 226 | is quite a bit slower, obviously.
|
|---|
| 227 |
|
|---|
| 228 | - chroot into the nfsroot: ``chroot /mnt/``
|
|---|
| 229 |
|
|---|
| 230 | - ``mount -o proc proc /proc``
|
|---|
| 231 |
|
|---|
| 232 | - ``mount -o sysfs sysfs /sys``
|
|---|
| 233 |
|
|---|
| 234 | - ``mkdir /mnt/testbox-tftp /mnt/testbox-backup``
|
|---|
| 235 |
|
|---|
| 236 | - Recreate ``/etc/fstab`` with::
|
|---|
| 237 |
|
|---|
| 238 | proc /proc proc defaults 0 0
|
|---|
| 239 | /dev/nfs / nfs defaults 1 1
|
|---|
| 240 | 10.42.1.1:/export/testbox-tftp /mnt/testbox-tftp nfs nfsvers=3 2 2
|
|---|
| 241 | 10.42.1.1:/export/testbox-backup /mnt/testbox-backup nfs nfsvers=3 3 3
|
|---|
| 242 |
|
|---|
| 243 | - Do ``mount /mnt/testbox-tftp && mount /mnt/testbox-backup`` to mount the
|
|---|
| 244 | two shares. This may be a good time to execute the instructions in the
|
|---|
| 245 | sections above relating to these two shares.
|
|---|
| 246 |
|
|---|
| 247 | - Edit ``/etc/initramfs-tools/initramfs.conf`` and change the ``MODULES``
|
|---|
| 248 | value from ``most`` to ``netboot``.
|
|---|
| 249 |
|
|---|
| 250 | - Append ``aufs`` to ``/etc/initramfs-tools/modules``. The advanced
|
|---|
| 251 | multi-layered unification filesystem (aufs) enables us to use a
|
|---|
| 252 | read-only NFS root. [3]_ [4]_ [5]_
|
|---|
| 253 |
|
|---|
| 254 | - Create ``/etc/initramfs-tools/scripts/init-bottom/00_aufs_init`` as
|
|---|
| 255 | an executable file with the following content::
|
|---|
| 256 |
|
|---|
| 257 | #!/bin/sh
|
|---|
| 258 | # Don't run during update-initramfs:
|
|---|
| 259 | case "$1" in
|
|---|
| 260 | prereqs)
|
|---|
| 261 | exit 0;
|
|---|
| 262 | ;;
|
|---|
| 263 | esac
|
|---|
| 264 |
|
|---|
| 265 | modprobe aufs
|
|---|
| 266 | mkdir -p /ro /rw /aufs
|
|---|
| 267 | mount -t tmpfs tmpfs /rw -o noatime,mode=0755
|
|---|
| 268 | mount --move $rootmnt /ro
|
|---|
| 269 | mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro
|
|---|
| 270 | mkdir -p /aufs/rw /aufs/ro
|
|---|
| 271 | mount --move /ro /aufs/ro
|
|---|
| 272 | mount --move /rw /aufs/rw
|
|---|
| 273 | mount --move /aufs /root
|
|---|
| 274 | exit 0
|
|---|
| 275 |
|
|---|
| 276 | - Update the init ramdisk: ``update-initramfs -u -k all``
|
|---|
| 277 |
|
|---|
| 278 | Note! It may be necessary to do ``mount -t tmpfs tmpfs /var/tmp`` to help
|
|---|
| 279 | this operation succeed.
|
|---|
| 280 |
|
|---|
| 281 | - Copy ``/boot`` to ``/mnt/testbox-tftp/maintenance-boot/``.
|
|---|
| 282 |
|
|---|
| 283 | - Copy the ``testbox-maintenance.sh`` file found in the same directory as this
|
|---|
| 284 | document to ``/root/scripts/`` (need to create the dir) and make it
|
|---|
| 285 | executable.
|
|---|
| 286 |
|
|---|
| 287 | - Create the systemd service file for the maintenance service as
|
|---|
| 288 | ``/etc/systemd/system/testbox-maintenance.service`` with the content::
|
|---|
| 289 |
|
|---|
| 290 | [Unit]
|
|---|
| 291 | Description=Testbox Maintenance
|
|---|
| 292 | After=network.target
|
|---|
| 293 | [email protected]
|
|---|
| 294 |
|
|---|
| 295 | [Service]
|
|---|
| 296 | Type=oneshot
|
|---|
| 297 | RemainAfterExit=True
|
|---|
| 298 | ExecStart=/root/scripts/testbox-maintenance.sh
|
|---|
| 299 | ExecStartPre=/bin/echo -e \033%G
|
|---|
| 300 | ExecReload=/bin/kill -HUP $MAINPID
|
|---|
| 301 | WorkingDirectory=/tmp
|
|---|
| 302 | Environment=TERM=xterm
|
|---|
| 303 | StandardOutput=journal+console
|
|---|
| 304 |
|
|---|
| 305 | [Install]
|
|---|
| 306 | WantedBy=multi-user.target
|
|---|
| 307 |
|
|---|
| 308 | - Enable our service: ``systemctl enable /etc/systemd/system/testbox-maintenance.service``
|
|---|
| 309 |
|
|---|
| 310 | - xxxx ... more ???
|
|---|
| 311 |
|
|---|
| 312 | - Before leaving the chroot, do ``mount /proc /sys /mnt/testbox-*``.
|
|---|
| 313 |
|
|---|
| 314 |
|
|---|
| 315 | - Testing the setup from a VM is kind of useful (if the nfs server can be
|
|---|
| 316 | convinced to accept root nfs mounts from non-privileged clinet ports):
|
|---|
| 317 |
|
|---|
| 318 | - Create a VM using the 64-bit debian profile. Let's call it "pxe-vm".
|
|---|
| 319 | - Mount the TFTP share somewhere, like M: or /mnt/testbox-tftp.
|
|---|
| 320 | - Reconfigure the NAT DHCP and TFTP bits::
|
|---|
| 321 |
|
|---|
| 322 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/AboveDriver NAT
|
|---|
| 323 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Action mergeconfig
|
|---|
| 324 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/TFTPPrefix M:/
|
|---|
| 325 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/BootFile pxelinux.0
|
|---|
| 326 |
|
|---|
| 327 | - Create the file ``testbox-tftp/pxelinux.cfg/0A00020F`` containing::
|
|---|
| 328 |
|
|---|
| 329 | PATH bios
|
|---|
| 330 | DEFAULT maintenance
|
|---|
| 331 | LABEL maintenance
|
|---|
| 332 | MENU LABEL Maintenance (NFS)
|
|---|
| 333 | KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
|
|---|
| 334 | APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 ro ip=dhcp aufs=tmpfs \
|
|---|
| 335 | boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/export/testbox-nfsroot
|
|---|
| 336 | LABEL local-boot
|
|---|
| 337 | LOCALBOOT
|
|---|
| 338 |
|
|---|
| 339 |
|
|---|
| 340 | -----
|
|---|
| 341 |
|
|---|
| 342 | .. [1] See http://www.syslinux.org/wiki/index.php?title=Category:Modules
|
|---|
| 343 | .. [2] See http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration
|
|---|
| 344 | .. [3] See https://en.wikipedia.org/wiki/Aufs
|
|---|
| 345 | .. [4] See http://shitwefoundout.com/wiki/Diskless_ubuntu
|
|---|
| 346 | .. [5] See http://debianaddict.com/2012/06/19/diskless-debian-linux-booting-via-dhcppxenfstftp/
|
|---|
| 347 |
|
|---|
| 348 |
|
|---|
| 349 | -----
|
|---|
| 350 |
|
|---|
| 351 | :Status: $Id: TestBoxImaging.txt 64523 2016-11-02 20:07:37Z vboxsync $
|
|---|
| 352 | :Copyright: Copyright (C) 2010-2016 Oracle Corporation.
|
|---|
| 353 |
|
|---|
| 354 |
|
|---|
