Stupid d-i tricks

Introduction

Debian Installer is an amazing piece of software, very extensible, and hackable. In addition to it's normal uses, it gives you a pretty nice environment running in a ramdisk that is easy to boot from the network, CD/DVD, usb drive, etc. This environment is really handy in a few different scenarios

• when you don't want to be booted from the system disk: for example if you are trying to copy it, delete it, do forensics, etc.
• using a system when you don't have a system disk, or don't have an OS installed on the system disk yet
• testing a system when using the system disk isn't ideal for some reason: for example if you don't want to disrupt the system disk, the kernel is too old, etc.

A lot of what d-i does is very useful in getting a ramdisk booted and setup properly, it sets up language settings, configures the network and proxy servers, etc. When booting d-i in normal "install" mode you can follow the menus up to the point where the disk partitioning starts, without causing any writes to disks in the machine. Once you are at this point you have a pretty nice environment setup and can then start using the shell for additional hacking.

While there are official proper ways to extend d-i with your own udebs and interfaces for adding menu entries, this page focuses on "hacks" that you can quickly do with any existing d-i. See the d-i wiki page for less hackish stuff.

Here are some cool things you can do.

Basics

• Getting files on the machine - wget

  d-i includes wget and you can use that to pull files onto the machine. After booting d-i, you can use the ui to get to the point where the network is configured, then get a shell (either from the main d-i menu, or a virtual console) and wget whatever you need. The files you pull will reside in the ramdisk, so your limited by the size of that, use df(1) to determine how much room you have. Also one thing to remember is that executables retrieved with wget are just copies, and will need to be chmod'd to be executable.

• Getting files off the machine - web server

  On really cool feature of d-i is that it includes a web server! Really it's just a simple shell script that uses netcat to speak some basic HTTP. To use it, after booting d-i proceed through the configure the network section, then go to the main menu and select the "Save debug logs" and then "web" options. You'll receive a notice that webserver is running and it's address, and you can point your web browser at the machine and get some useful information. If you want to make another file available, drop to a shell and put the file (or command output, etc) in /var/log/ and then it will show up. If you want to see how th web server is implemented, look at the /usr/bin/httpd shell script.

• Adding programs to d-i by hand

  Using the wget method described above you can pull additional programs on to the system. The d-i environment is pretty limited and only provides a few system libraries, so you might need grab some libraries as well. Here is a set of typical steps.

  1. On a system of the same arch, grab the program you want and make it available via http.
  2. We need to provide any library dependencies that the program needs. These can be gathered from same system you got the program from
     • Method #1:
       • run ldd on the binary and determine if there are any library dependencies that aren't provided in the d-i environment. Make those available as well. For example if we wanted to determine what libraries the "sl" program uses on an amd64 system

         $ ldd /usr/bin/sl
                 libncurses.so.5 => /lib/libncurses.so.5 (0x00002ad572d2c000)
                 libc.so.6 => /lib/libc.so.6 (0x00002ad572e87000)
                 libdl.so.2 => /lib/libdl.so.2 (0x00002ad5730c4000)
                 /lib64/ld-linux-x86-64.so.2 (0x00002ad572c14000)
         

         d-i provides all of those except for libncurses
       • On the d-i booted system, use the wget method to get the binary and needed libraries on the system.
       • chmod +x the binary
       • Put the binary in the system PATH, ie /usr/bin
       • Put the libraries in the library path, ie /usr/lib
       • The libraries need the same symlinks as they would on a normal system, since that's what the binaries will refer to. You can create these by hand, or if you tar up the things you need on the system providing the tools, include the symlinks in the tarball too
       • Once everything is installed, try to run the binary. You might find that you missed one or one of the libraries itself had a dependency. Sometimes this is a iterative process, keep grabbing stuff until you get things to work. For example, continuing with the example above, without the ncurses library we'd get an error like

         # sl
         sl: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory
         

         Once I've figured out all the things needed to get something working, I keep a list suitable for use with tar so if I need to do it again I can generate a current tarball of everything, wget it to the d-i system, and untar it.

     • Method #2: dannf points out the mklibs-copy tool/package can do the above job for you and does a better job of getting only the libraries that the program actually needs, which is what d-i itself uses.
     
     
• Installing additional udebs

  If the additional program you want is available as a udeb, you can use the anna-install program to install it. There are only a few things available as udebs that aren't already loaded in d-i by default, but there are a couple useful things.

• Installing additional debs

  You can use udpkg to install normal debian packages and it mostly works. Depending on which d-i image you used, you might even have a partial archive full of debs available that you can refer to directly like

  udpkg -i /cdrom/pool/main/s/sl*.deb
  

• Utilizing an existing root

  If you have a root disk mounted, sometimes it's handy to be able to utilize the full install for things. d-i provides the chroot command, so you can run things in the system root that way, or you can just take advantage of libraries and binaries with something like

  LD_LIBRARY_PATH=/target/lib:/target/usr/lib /target/usr/bin/foo
  

• Running multiple programs

  On i386 and amd64 at least, d-i runs on VC #1, but can also run shells on VC #2 and #3. By selecting the "Execute a shell" option in the d-i interface on VC #1, you have a total of 3 shells that you can run things in. If you need more shells than that you can add addition virtual consoles. To do that run something like

  echo "tty5::askfirst:-/bin/sh" >> /etc/inittab
  kill -HUP 1
  

Now we have a way to get files on and off the system, add additional programs, take advantage of the system root, and do multiple things at the same time.

Tricks

• Cleaning a hard drive

  Before I recycle/reuse hard drives, I like to wipe the existing data off of them so I can be sure not to lose any private data/passwords/etc. Wiping the data from a disk requires that you are not booted from the drive at the time, so this is good use for d-i. The shred(1) from the coreutils package doesn't need any shared libraries beyond what's provided by d-i. Use the wget method to get it and then run something like

  # shred -u -v -n 10 /dev/sda
  

  You can run several of these processes in parallel with no problems (using the "multiple" technique described above), I often set up a machine to clean stacks of disks in this way before I send them to the computer recycling/reuse center. Read the shred(1) manpage for more info. You might also consider using the wipe(1) command from the wipe package, it also has no additional library dependencies. It also has a more entertaining man page :)

• Testing a hard drive

  You can test hard drives using the badblocks(8) command. Where shred is about making sure the data on the drive is overwritten, badblocks is for testing that the blocks are working correctly. This is a good idea to do on new drives to ensure they are working ok during their RMA or warranty period, or also any time you are about to redeploy a used drive in a new purpose. badblocks can be used in several different modes: read-only (default), non-destructive read/write test where the block contents are saved before and then restored after the write test, and a destructive read/write test. badblocks is already part of d-i, so just get a shell and run

  read-only

  # badblocks -s -v -b 4096 -c 10240 /dev/sda
  

  non-destructive read/write

  # badblocks -s -v -n -b 4096 -c 10240 /dev/sda
  

  destructive read/write

  # badblocks -s -v -w -b 4096 -c 10240 /dev/sda
  

  In the above examples -s is status, -v is verbose, -b is the block size which we increase to 4k from the 1k default, and -c is the count of blocks to test at a time which we increase a lot from the default of 64 in order to speed things up since modern systems have plenty of RAM. If you wanted to test the disk more than just once or as an extended stress test, you can add a -p # option to specify a number of passes. Also note that you can run multiple badblocks at the same time, which is a particularly good way of exercising the system. NOTE: The destructive write test is also results in a clean the drive like the above shred example.

• S.M.A.R.T. testing a hard drive

  For drives that support S.M.A.R.T. you can run the drive's smart tests using the smartctl utility. Use the wget method to grab the following from a full system of the same architecture

  /usr/sbin/smartctl
  /usr/lib/libstdc++.so.6.0.10 (or whatever)
  /usr/lib/libstdc++.so.6 -> /usr/lib/libstdc++.6.0.10 symlink
  /lib/libgcc_s.so.1
  

  Query the drive info (for a sata drive in these examples, needs -d ata)

  # smartctl -a -d ata /dev/sda
  

  Query the drive capabilities

  # smartctl -c -d ata /dev/sda
  

  Check the health of the drive

  # smartctl -H -d ata /dev/sda
  

  Read the error log

  # smartctl -l error -d ata /dev/sda
  

  Read the self test log

  # smartctl -l selftest -d ata /dev/sda
  

  Run the full offline test (which you can query the state of using the above)

  # smartctl -t offline -d ata /dev/sda
  

  Run the long test

  # smartctl -t long -d ata /dev/sda
  

  When testing a new drive you probably want to do something like

  1. use -a to read about the drive and confirm its the drive you think it is :)
  2. use -c to read what capabilites the drive has
  3. use -H to confirm the health of the disk is OK
  4. use -l error to confirm there are no errors.
  5. run the long test with -t long
  6. check the status of the running long test with -c
  7. after the test completes, check the error log with -l error and health with -H to confirm that things are OK

  Probably in addition to looking for really bad failures, for these tests to be useful it might be good to record the drive counters before and after the tests to see what changed.

  Here is some more info about SMART.

• Upgrading firmware

  Some systems (HP ProLiant, and Dell server for example) have Linux utilities for upgrading firmware. You can boot d-i, wget the update utility, and update the firmware on the system before installing. This might be particularly useful if the firmware upgrade is required in order to enable a piece of hardware that's needed for the install, when it would not be possible to install the system first and then upgrade the firmware.

  This method would also probably work for any diagnostic utilities that run under linux.

• Gathering system data

  If you've played with the built in webserver, you know that makes some information available about what d-i found on the system, including lspci output. This can be pretty useful when installing, especially if d-i fails to install on newer hardware. You can often plug some of the lspci output into google and find others that are working on the same problem. But one problem with this is that the database lspci uses to name the devices is static and built into d-i at the time of release. This often means it won't have names for the newer hardware you are working with.

  From another system, grab /usr/bin/update-pciids and its dependency /usr/bin/which and make them available via http. Use the wget method, install in /usr/bin, and chmod +x them. Then run update-pciids

  # update-pciids
  Connecting to pciids.sourceforge.net[66.35.250.209]:80
  
  pci.ids.new          100% |*****************************|   494 KB    00:00 ETA
  Done.
  

  Then run something like

  # lspci -nnv >/var/log/new-lspci
  

  and retrieve the updated output via the webserver.

• Running an install remotely

  This is actually a feature of d-i, no tricks involved. Boot d-i in expert mode, proceed with the steps (which will include setting up the network) until you get to select additional d-i components to load. Select the "remote install via ssh" module, after it loads select that option and follow the instructions to set a password for the "installer" user. The system generates an ssh host key and starts ssh and then you can login remotely and run the install. You want use a normal 80x24 terminal window and not resize the window as that can disrupt the d-i interface.

  I have used this feature when I was setting up a system for someone who was 3000 miles away and I wanted to let them do the install so they could choose partition details, set usernames, passwords, crypto passphrases, and other sensitive information. Very handy.

• Cloning another system via rsync/ssh
  1. On the remote machine you intend to clone do the following
     • Edit /etc/ssh/sshd_config and enable root logins temporarily, and either allow password logins or setup an ssh key for root. Restart ssh so the changes take effect.
     • You want as little running on the remote system as possible, so you don't have to worry about copying any running state in the filesystem. While most services should be designed to be able to handle encountering weird situations like this, it's probably better to avoid the situation if possible. Boot the remote machine single user, and then start ssh by hand.
  2. Now, boot d-i on the target system, and proceed through the menus to setup the network, etc. You can even use the disk partitioning to get the disks partitioned how you like and filesystems created. Stop before doing the base install.
  3. Next we need to get ssh and rsync on to the system.
     • Method #1: Here is a list of the files needed to get ssh and rsync working. On a full system of the same arch, use the list to create a tarball.

       $ tar zcvf rsync-ssh.tar.gz `cat rsync-ssh-list.txt`
       

       Make that tarball available via http, like in a public_html directory for example. Now drop to a shell and use wget to get the tarball on the system, and untar it in the root of the ramdisk. Test and make sure ssh and rsync run properly.
     • Method #2: There is an openssh-client-udeb available, install it with

       # anna-install openssh-client-udeb
       

       Then you need to get rsync, libacl, libattr, and libpopt. Here's a list. Similar to above generate a tarball of the needed stuff and wget it to the system.
  4. If you used d-i to partition and create filesystems, they should already be mounted under /target, if not mount them there.
  5. From the /target directory, use rsync to copy the remote system to the target system. I use something like this

     # rsync -avzWHS --dry-run -x --delete --numeric-ids -P root@orighost:/ .
     

     Here is an explanation of the options:

     a = archive, which is a short way to specify all the flags that preserve the files like you'd want in an archive of the filesystem
     v = verbose
     z = compress data before sending, you might not use this if both systems are on the same LAN, but on a WAN it helps
     W = sends the Whole file
     H = recreate hardlinks
     S = handle Sparse files correctly
     x = don't cross filesystem boundaries, this is so we don't end up copying dynamic filesystems like /proc. But this means you'll need to run an rsync for each filesystem on the original system that needs to be moved over. Alternately you could use the --exclude option to prevent grabbing dynamic filesystems.
     numeric-ids = copy using numeric UIDs. The d-i system you're running under doesn't have the same users as the original system. In the case where it doesn't have the user it will fall back to numeric UID, but in cases where it does, you might end up with a mess. Use numeric-ids to be sure it does the right thing.
     P = short for progress+partial. progress gives you nice progress feedback during the transfer, so you know how fast it's going and that it's not hung. partial says to keep partial files, which is probably contradictory when we're using W.
     dry-run = don't do anything, once you're sure it going to do the right thing, then rerun with that removed
     
     
     
  6. After running the above without dry-run and the copy completes, run it again. There should either be nothing to transfer or just a few files on the original system that are changing such as log files. You can repeat the command multiple times until you're sure that you have everything.
  7. Now that you have a copy of the original system, there are a few things to consider: How is the hardware of the new system different from the old system? What needs to change for the new system to work? While you have the new system mounted under d-i at /target, make your changes now while it's easy. You can even use "chroot /target" to put you into the new system where you have a more complete set of tools than in d-i. When I have cloned systems in this manner, here are some of the things I have needed to change
     • Adjust /etc/fstab for any filesystem, device, etc changes.
     • udev has a "feature" where it assigns eth* names to ethernet devices and records the association of name to MAC in /etc/udev/rules.d/z25_persistent-net.rules. The new system most likely has different network interfaces, so if you want to use the same names, you need to edit that file and delete the old entries.
     • Adjust /etc/network/interfaces to reflect any network changes
     • If the new system is also going to have a new IP address, find anywhere it might be hard coded and fix it
     The above can be an iterative process as you discover things that need to be changed, but keep working on it, you'll get everything working.
  8. Next we need to install a boot loader on new system. Presumably the original system had a bootloader installed already, and its config files were on the filesystem and copied over to the new disk. There are two steps needed
     1. If the hardware has changed, the bootloader config files need to be adjusted in order to refer to the new devices and partitions. If you are using i386 or amd64, this probably means grub, and adjusting the /boot/grub/menu.lst and /boot/grub/device.map files. If you're on another arch this step is left as an exercise for the reader :)
     2. The bootloader needs to be installed on new drive (not all arch bootloaders need this, but most do). In the case of i386 and amd64, this means grub's primary stage bootloader installed in the Master Boot Record. There are multiple ways to do this, using grub interactively, install-grub, getting the system to boot via d-i rescue mode, maybe the d-i grub menu option. When I did this it was on a particularly tricky system, since I was using a cciss device that wasn't detected by grub when booted via d-i, so I don't have good generic notes. If I do it again on a less weird system I will update with more detailed instructions (or if someone wants to send me some...).
     3. Once you are sure that the filesystems are all copied over, reconfigured properly, and the bootloader updated and installed properly, reboot the system.
        • If the bootloader fails to load, you can always boot d-i again via the normal method, mount the disks, and try to fix the bootloader, or boot it via the "rescue" method to try and boot the actual kernel/initramfs on the root fs and then fix the bootloader.
        • If the bootloader loads but fails to boot the kernel, you probably just forgot to change the config to point to the right partition
        • If the kernel starts booting, but then fails to find a root, you just need to adjust the bootloader to fix the root= passed to the kernel
        • If the kernel starts booting and finds a root, but there are issues starting services, you probably need config changes. If the system manages to boot without those services and you can get a root shell and fix them, great. If not then next time boot with "init=/bin/sh" and edit the config files and reboot.
  9. Once you are done, don't forget to turn ssh root logins off on the original system.

Other ideas

• System burn-in with things like cpuburn, badblocks etc. UPDATE: badblocks example above, and there is now a cpuburn udeb but I haven't tried it
• Installing strace to debug something UPDATE: this works and combined with the next item is a good way to test something and get results off the machine
• use script(1) to record a log of something and then put it in /var/log to retrieve via the webserver
• An easily bootable SETI@Home, Boinc, etc. client.
• A simple website or mirror. If you booted from CD/DVD debian media, creating a debian mirror should be pretty easy.
• A simple bittorrent seeder.
• Simple network analysis
• Playing moon-buggy or using sl, both only require libncurses. UPDATE: tried it, it works :)
• The "bb" demo seems to require a bunch of extra libraries (including X Windows stuff), so I haven't tried it.

Thanks to Colin Watson and dann frazier for contributing comments and tricks.