VirtualBox

source: vbox/trunk/src/VBox/ValidationKit/docs/TestBoxImaging.txt@ 64523

最後變更 在這個檔案從64523是 64523,由 vboxsync 提交於 8 年 前

TestBoxImaging: Updates & service script.

  • 屬性 svn:eol-style 設為 native
  • 屬性 svn:keywords 設為 Author Date Id Revision
檔案大小: 12.5 KB
 
1
2Testbox Imaging (Backup / Restore)
3==================================
4
5
6Introduction
7------------
8
9This document is explores deloying a very simple drive imaging solution to help
10avoid needing to manually reinstall testboxes when a disk goes bust or the OS
11install seems to be corrupted.
12
13
14Definitions / Glossary
15======================
16
17See AutomaticTestingRevamp.txt.
18
19
20Objectives
21==========
22
23 - Off site, no admin interaction (no need for ILOM or similar).
24 - OS independent.
25 - Space and bandwidth efficient.
26 - As automatic as possible.
27 - Logging.
28
29
30Overview of the Solution
31========================
32
33Here is a brief summary:
34
35 - Always boot testboxes via PXE using PXELINUX.
36 - Default configuration is local boot (hard disk / SSD)
37 - Restore/backup action triggered by machine specific PXE config.
38 - Boots special debian maintenance install off NFS.
39 - A maintenance service (systemd style) does the work.
40 - The service reads action from TFTP location and performs it.
41 - When done the service removes the TFTP machine specific config
42 and reboots the system.
43
44Maintenance actions are:
45 - backup
46 - backup-again
47 - restore
48 - refresh-info
49 - rescue
50
51Possible modifier that indicates a subset of disk on testboxes with other OSes
52installed. Support for partition level backup/restore is not explored here.
53
54
55How to use
56----------
57
58To perform one of the above maintenance actions on a testbox copy the
59
60
61Storage Server
62==============
63
64The storage server will have three areas used here. Using NFS for all three
65avoids extra work getting CIFS sharing right too (NFS is already a pain).
66
67 1. /export/testbox-tftp - TFTP config area. Read-write.
68 2. /export/testbox-backup - Images and logs. Read-write.
69 3. /export/testbox-nfsroot - Custom debian. Read-only, no root squash.
70
71
72TFTP (/export/testbox-tftp)
73============================
74
75The testbox-tftp share needs to be writable, root squashing is okay.
76
77We need files from both PXELINUX and SYSLINUX to make this work now. On a
78debian system, the ``pxelinux`` and ``syslinux`` packages needs to be
79installed. We actually do this further down when setting up the nfsroot, so
80it's possible to get them from there by postponing this step a little. On
81debian 8.6.0 the PXELINUX files are found in ``/usr/lib/PXELINUX`` and the
82SYSLINUX ones in ``/usr/lib/syslinux``.
83
84The initial PXE image as well as associated modules comes in three variants,
85BIOS, 32-bit EFI and 64-bit EFI. We'll only need the BIOS one for now.
86Perform the following copy operations::
87
88 cp /usr/lib/PXELINUX/pxelinux.0 /mnt/testbox-tftp/
89 cp /usr/lib/syslinux/modules/*/ldlinux.* /mnt/testbox-tftp/
90 cp -R /usr/lib/syslinux/modules/bios /mnt/testbox-tftp/
91 cp -R /usr/lib/syslinux/modules/efi32 /mnt/testbox-tftp/
92 cp -R /usr/lib/syslinux/modules/efi64 /mnt/testbox-tftp/
93
94
95For simplicitly, all the testboxes boot using good old fashioned BIOS, no EFI.
96However, it doesn't really hurt to be prepared.
97
98The PXELINUX related files goes in the root of the testbox-tftp share. (As
99mentioned further down, these can be installed on a debian system by running
100``apt-get install pxelinux syslinux``.) We need the ``*pxelinux.0`` files
101typically found in ``/usr/lib/PXELINUX/`` on debian systems (recent ones
102anyway). It is possible we may need one ore more fo the modules [1]_ that
103ships with PXELINUX/SYSLINUX, so do copy ``/usr/lib/syslinux/modules`` to
104``testbox-tftp/modules`` as well.
105
106
107The directory layout related to the configuration files is dictated by the
108PXELINUX configuration file searching algorithm [2]_. Create a subdirectory
109``pxelinux.cfg/`` under ``testbox-tftp`` and create the world readable file
110``default`` with the following content::
111
112 PATH bios
113 DEFAULT local-boot
114 LABEL local-boot
115 LOCALBOOT
116
117This will make the default behavior to boot the local disk system.
118
119Create ``pxelinux.cfg/do-backup``, ``pxelinux.cfg/do-backup-again``,
120``pxelinux.cfg/do-restore``, ``pxelinux.cfg/do-refresh-info``, and
121``pxelinux.cfg/do-rescue`` configuration files on the form::
122
123 PATH bios
124 DEFAULT maintenance
125 LABEL maintenance
126 MENU LABEL Maintenance (NFS)
127 KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
128 APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 testbox-action-backup ro ip=dhcp aufs=tmpfs boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/export/testbox-nfsroot,ro nfsvers=3 nfsrootdebug
129 LABEL local-boot
130 LOCALBOOT
131
132When you want to preform an action on a testbox, copy the ``do-<action>`` to
133``pxeclient.cfg/<HEX-ip-addr>`` and trigger a boot of the testbox. The machine
134config will be removed automatically once the action has been successfully
135completed.
136
137
138
139Images and logs (/export/testbox-backup)
140=========================================
141
142The testbox-backup share needs to be writable, root squashing is okay.
143
144In the root there must be a file ``testbox-backup`` so we can easily tell
145whether we've actually mounted the share or are just staring at an empty mount
146point directory.
147
148The ``testbox-maintenance.sh`` script maintains a global log in the root
149directory that's called ``maintenance.log``. Errors will be logged there as
150well as a ping and the action.
151
152We use a directory layout based on dotted decimal IP addresses here, so for a
153server with the IP 10.40.41.42 all its file will be under ``10.40.41.42/``:
154
155``<hostname>``
156 The name of the testbox (empty file). Help finding a testbox by name.
157
158``testbox-info.txt``
159 Information about the testbox. Starting off with the name, decimal IP,
160 PXELINUX style hexadecimal IP, and more.
161
162``maintenance.log``
163 Maintenance log file recording what the maintenance service does.
164
165``disk-devices.lst``
166 Optional list of disk devices to consider backuping up or restoring. This is
167 intended for testboxes with additional disks that are used for other purposes
168 and should touched.
169
170``sda.raw.gz``
171 The gzipped raw copy of the sda device of the testbox.
172
173``sd[bcdefgh].raw.gz``
174 The gzipped raw copy sdb, sdc, sde, sdf, sdg, sdh, etc if any of them exists
175 and are disks/SSDs.
176
177
178Note! If it turns out we can be certain to get a valid host name, we might just
179 switch to use the hostname as the directory name instead of the IP.
180
181
182Debian NFS root (/export/testbox-nfsroot)
183==========================================
184
185The testbox-nfsroot share should be read-only and must **not** have root
186squashing enabled.
187
188There are several ways of creating a debian nfsroot, but since we've got a
189tool like VirtualBox around we've just installed it in a VM, prepared it,
190and copied it onto the NFS server share.
191
192As of writing debian 8.6.0 is current, so a minimal 64-bit install of it was
193done in a VM. After installation the following modifications was done:
194
195 - ``apt-get install pxelinux syslinux initramfs-tools zip gddrescue joe``
196 and optionally ``apt-get install smbclient cifs-utils``.
197
198
199 - ``/etc/default/grub`` was modified to set ``GRUB_CMDLINE_LINUX_DEFAULT`` to
200 ``""`` instead of ``"quiet"``. This allows us to see messages during boot
201 and perhaps spot why something doesn't work on a testbox. Regenerate the
202 grub configuration file by running ``update-grub`` afterwards.
203
204 - Create the directory ``/etc/systemd/system/[email protected]`` and create
205 the file ``noclear.conf`` in it with the following content::
206
207 [Service]
208 TTYVTDisallocate=no
209
210 This stops getty from clearing VT1 and let us see the tail of the boot up
211 messages, which includes messages from the testbox-maintenance service.
212
213 - Mount the testbox-nfsroot under ``/mnt/`` with write privileges. (The write
214 privileges are temporary - don't forget to remove them later on.)::
215
216 mount -t nfs myserver.com:/export/testbox-nfsroot
217
218 Note! Adding ``-o nfsvers=3`` may help with some NTFv4 servers.
219
220 - Copy the debian root and dev file system onto nfsroot. If you have ssh
221 access to the NFS server, the quickest way to do it is to use ``tar``::
222
223 tar -cz --one-file-system -f /mnt/testbox-maintenance-nfsroot.tar.gz . dev/
224
225 An alternative is ``cp -ax . /mnt/. && cp -ax dev/. /mnt/dev/.`` but this
226 is quite a bit slower, obviously.
227
228 - chroot into the nfsroot: ``chroot /mnt/``
229
230 - ``mount -o proc proc /proc``
231
232 - ``mount -o sysfs sysfs /sys``
233
234 - ``mkdir /mnt/testbox-tftp /mnt/testbox-backup``
235
236 - Recreate ``/etc/fstab`` with::
237
238 proc /proc proc defaults 0 0
239 /dev/nfs / nfs defaults 1 1
240 10.42.1.1:/export/testbox-tftp /mnt/testbox-tftp nfs nfsvers=3 2 2
241 10.42.1.1:/export/testbox-backup /mnt/testbox-backup nfs nfsvers=3 3 3
242
243 - Do ``mount /mnt/testbox-tftp && mount /mnt/testbox-backup`` to mount the
244 two shares. This may be a good time to execute the instructions in the
245 sections above relating to these two shares.
246
247 - Edit ``/etc/initramfs-tools/initramfs.conf`` and change the ``MODULES``
248 value from ``most`` to ``netboot``.
249
250 - Append ``aufs`` to ``/etc/initramfs-tools/modules``. The advanced
251 multi-layered unification filesystem (aufs) enables us to use a
252 read-only NFS root. [3]_ [4]_ [5]_
253
254 - Create ``/etc/initramfs-tools/scripts/init-bottom/00_aufs_init`` as
255 an executable file with the following content::
256
257 #!/bin/sh
258 # Don't run during update-initramfs:
259 case "$1" in
260 prereqs)
261 exit 0;
262 ;;
263 esac
264
265 modprobe aufs
266 mkdir -p /ro /rw /aufs
267 mount -t tmpfs tmpfs /rw -o noatime,mode=0755
268 mount --move $rootmnt /ro
269 mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro
270 mkdir -p /aufs/rw /aufs/ro
271 mount --move /ro /aufs/ro
272 mount --move /rw /aufs/rw
273 mount --move /aufs /root
274 exit 0
275
276 - Update the init ramdisk: ``update-initramfs -u -k all``
277
278 Note! It may be necessary to do ``mount -t tmpfs tmpfs /var/tmp`` to help
279 this operation succeed.
280
281 - Copy ``/boot`` to ``/mnt/testbox-tftp/maintenance-boot/``.
282
283 - Copy the ``testbox-maintenance.sh`` file found in the same directory as this
284 document to ``/root/scripts/`` (need to create the dir) and make it
285 executable.
286
287 - Create the systemd service file for the maintenance service as
288 ``/etc/systemd/system/testbox-maintenance.service`` with the content::
289
290 [Unit]
291 Description=Testbox Maintenance
292 After=network.target
293 [email protected]
294
295 [Service]
296 Type=oneshot
297 RemainAfterExit=True
298 ExecStart=/root/scripts/testbox-maintenance.sh
299 ExecStartPre=/bin/echo -e \033%G
300 ExecReload=/bin/kill -HUP $MAINPID
301 WorkingDirectory=/tmp
302 Environment=TERM=xterm
303 StandardOutput=journal+console
304
305 [Install]
306 WantedBy=multi-user.target
307
308 - Enable our service: ``systemctl enable /etc/systemd/system/testbox-maintenance.service``
309
310 - xxxx ... more ???
311
312 - Before leaving the chroot, do ``mount /proc /sys /mnt/testbox-*``.
313
314
315 - Testing the setup from a VM is kind of useful (if the nfs server can be
316 convinced to accept root nfs mounts from non-privileged clinet ports):
317
318 - Create a VM using the 64-bit debian profile. Let's call it "pxe-vm".
319 - Mount the TFTP share somewhere, like M: or /mnt/testbox-tftp.
320 - Reconfigure the NAT DHCP and TFTP bits::
321
322 VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/AboveDriver NAT
323 VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Action mergeconfig
324 VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/TFTPPrefix M:/
325 VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/BootFile pxelinux.0
326
327 - Create the file ``testbox-tftp/pxelinux.cfg/0A00020F`` containing::
328
329 PATH bios
330 DEFAULT maintenance
331 LABEL maintenance
332 MENU LABEL Maintenance (NFS)
333 KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
334 APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 ro ip=dhcp aufs=tmpfs \
335 boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/export/testbox-nfsroot
336 LABEL local-boot
337 LOCALBOOT
338
339
340-----
341
342.. [1] See http://www.syslinux.org/wiki/index.php?title=Category:Modules
343.. [2] See http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration
344.. [3] See https://en.wikipedia.org/wiki/Aufs
345.. [4] See http://shitwefoundout.com/wiki/Diskless_ubuntu
346.. [5] See http://debianaddict.com/2012/06/19/diskless-debian-linux-booting-via-dhcppxenfstftp/
347
348
349-----
350
351:Status: $Id: TestBoxImaging.txt 64523 2016-11-02 20:07:37Z vboxsync $
352:Copyright: Copyright (C) 2010-2016 Oracle Corporation.
353
354
注意: 瀏覽 TracBrowser 來幫助您使用儲存庫瀏覽器

© 2025 Oracle Support Privacy / Do Not Sell My Info Terms of Use Trademark Policy Automated Access Etiquette