|Revision 0.1||2005-05-06||Revised by: taggart|
|merged existing firmware and iLO howtos and add a bunch of stuff|
HP sells "PC servers" that use Intel (ia32 and em64t) and AMD (amd64) processors under the "ProLiant" brand name. General information on these systems is available at http://www.hp.com/go/proliant.
Linux is one of the operating systems officially supported by HP, but only releases from Red Hat and Novell. Historically HP has only supported propriatary operating systems on these systems and this is visible in their approach to Linux support as well. Drivers, remote management devices, and monitoring software are all provided with distribution specific instructions and packaging. Some tasks even require using Microsoft Windows to perform.
This document is intended to provide the necessary information to work around these restrictions and effectively use ProLiant systems with Debian. With luck it provides enough information for users of other unsupported operating systems to get their systems working as well. It is clear that HP as a business can't officially support all operating systems, but that doesn't mean users shouldn't be able to support themselves.
This document also tries to minimize duplication of information already available elsewhere, while still providing the important details all in one place for convenience. When possible this document will defer to documentation from HP or Debian.
This document, Using HP's integrated Lights Out (iLO) with Debian,
is copyrighted (c) 2004-2005 by Matt Taggart
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Linux is a registered trademark of Linus Torvalds. Debian is a registered trademark of Software in the Public Interest.
Before installing any version of Debian there are some things you need to consider and setup.
If your system contains a RAID controller you'll need to decide if/how you want to use it. If you choose to use it you'll need to configure the array. In some systems it is possible to disconnect the drives from the RAID controller and connect them to an on board non-RAID controller if you prefer that.
If your system contains an HP SmartArray controller, on boot it will initialize the controller and print something like, HP Smart Array 5i Controller Initializing. When you see that message you can press F8 to enter the setup utility. You'll need to setup at least one logical drive containing one or more physiscal drives. Depending on the number of drives it's possible to setup multiple logical drives, different RAID levels with various performance setting, hot spare drives, etc. Configuring drive arrays is a very controversial topic, everyone has their favorite settings. This guide doesn't attempt to explain, ask an expert if needed.
All recent ProLiants have an onboard system BIOS which you can enter after post and adjust settings (some older models booted a special diagnostics software that lived on a special disk partition). The usual way to enter the BIOS is to hit F10 when prompted during boot up sequence. There are several options you might want to change.
On some models the MAC addresses of the onboard network adapters are displayed, you probably want to make a note of them for use in your DHCP server config or for PXE netbooting. FIXME: add a reference to that section once we add it.
The BIOS has an OS Selection option that you should set to Linux. On the dl360g3/dl380g3 at least this is under the System Options menu. This supposedly changes settings in the BIOS to be optimized for Linux, but it's unknown what it actually does. This is assumed to be the most tested/supported setting, so it's probably a good idea to set it.
Under the Advanced Options menu (at least on the systems mentioned above) there is a POST F1 Prompt. This controls what the system does right after post when it presents you with F1 to continue prompt, either wait for the key press, wait for a few seconds before timing out and proceeding, or not to prompt at all. You need to decide what you want this behaviour to be. Consider,
You may want the system to be able to boot unattended, for example to recover automatically from a power failure.
You might want to be aware that the system has rebooted and want to be present to watch the boot, for example if you are using crypto filesystems that require you to type in a passphrase before booting.
There may be other options you want to change, consult the HP documentation or your local expert for more info.
If you are installing via the iLO's text console you need to be aware that things that put the system in graphics mode will prevent you from using the text console. This includes boot loader framebuffer based spash screens, Linux kernel framebuffer mode, and the X Windows System. In a lot of cases you will need to boot with special options, or use special install media to prevent this sort of thing from happening. More specific notes are mentioned in the sections below.
Installing woody on older ProLiants is fairly straight forward, but systems such as the dl360g3/dl380g3 and newer have a newer SmartArray controller that was not supported by the install kernel in woody (2.4.18). In addition that kernel lacks the tg3 driver needed for their lan devices, which can be worked around using removable media, but is generally difficult. To overcome these problems several people have produced custom boot-floppies with the correct set of drivers to make installing on these systems easier.
ProLiant specific, with installation and creation instructions: http://people.debian.org/~taggart/boot-floppies/
Dell specific but more up to date and may work depending on your system: http://wiki.osuosl.org/display/LNX/Debian+on+Dell+Servers
As mentioned in the general section, if using the iLO text console to install you need tell the installer not to use framebuffer mode. The above boot-floppies are built with framebuffer mode turned off, but if you are using generic woody boot-floppies or another variant you may have to turn off the framebuffer mode. On initial boot the syslinux boot loader presents a screen that lists several function keys you can press for help and additional boot options. On one of those pages there are instructions on how boot with the framebuffer disabled by appending "video=vga16:off" to your boot selection. You will need to remember to make sure that the bootloader installed also either had this option or that you run a kernel with framebuffer disabled.
After the initial install, you will most likely want to upgrade the kernel to a more recent kernel, backported to woody. http://backports.org maintains some backports but doesn't always have the latest since there is not much demand. Once sarge releases I suspect they might stop backporting for woody unless there are people willing to contribute.
If you are interested in using grub and are using a SmartArray controller, be aware that there was a problem with the version of grub in woody that requires a work-around. After installing grub (and before rebooting), add "(hd0) /dev/cciss/c0d0" to /boot/grub/device-map and rerun grub-installer.
sarge and newer use the new Debian-Installer http://www.debian.org/devel/debian-installer/. FIXME: explain using d-i including pxe booting
As mentioned in the general section, if using the iLO text console to install you need tell the installer not to use framebuffer mode. The isolinux bootloader used by the new debian-installer uses a framebuffer spashscreen to print a fancy Debian graphic and the words "Press F1 to continue". This causes the system to go into graphics mode, and then you can't see anything on the iLO text console. Fortunately you can still type and you can blindly hit F1 and d-i proceeds to the next screen which it back in text mode. Next when you boot you need to disable the framebuffer (how to do so is explained on one of the help screens) by appending "debian-installer/framebuffer=false" to your boot selection. You will want the bootloader installed on the system to have this option as well, and fortunately d-i take care of this for you automatically.
While HP does not currently support Debian, it is possible to use what they make available to keep your system running well. HP organizes support downloads based on what supported O/S you are running. Since there is no way to select Debian, we can still get to things we need by selecting the latest version of SuSE or Red Hat. To get there go to HP's support site and browse to find your model and select 'download drivers and software', and then pick something like 'SUSE LINUX Enterprise Server 9 (x86)'. Once on that page there are various categories of stuff available.
This is some general info about how to deal with stuff from HP. More specific details will be documented in later sections of this document.
The firmware images are usually in something HP calls an 'scexe' file, which is a binary blob wrapped by a posix shell script. When invoked, this 'scexe' unpacks the binary blob of files and then runs a binary from the blob, usually an installer. This binary often does things like try to determine what distro and version it's running on. In some cases you can get this to work on Debian, but in general this usually breaks horribly. If it does work the 'scexe' scripts also support an '--unpack=' flag, which allows you to do the unpack step without invoking the installer as well. In some cases the stuff unpacked gives you what you need to upgrade. More details in the firmware section below.
While HP makes a point of providing drivers on the support page, these are always custom built for the non-Debian distro in question. Fortunately, HP is good about making sure that the HP specific driver work that they do is also sent upstream to kernel.org. So use a Debian or upstream kernel and you'll be fine.
Since you're running Debian, a lot of the software listed probably won't be of interest to you. It's all propriatary and most of it is specific to particular product support options or supported distros, and so doesn't make sense for Debian. However, there are a few tools that can be made to work in Debian if you have a need for them, mainly the management tools.
HP provides various utilities to help in managing the server, for example health monitoring, array config/monitoring, etc. These are almost always distributed as RPM packages. You can use alien(1) to convert them, but they usually have RPM install scripts that do various things on install (mostly check what distro you're on and setup config files). While sometimes you can use 'alien -c' to convert these scripts, often the scripts will have Red Hat or SuSE'isms in them, and will fail on install. To work around this you can either not convert the script(no '-c') and then manually do the needed steps that the scripts would have done, or you can use 'alien -g -c' to create an intermediate debian source package where you can go in and clean up the install scripts. Some of these methods will be documented in later section of this document.
HP provides a service for notifiying system owners when new versions of items are available. You can go to https://h30046.www3.hp.com/subprofile.php?SUBS=DRIVERSIGNUP and register for the service, it's free. They do require you to fill out a form but it doesn't require much personal info other than your name and email address.
In addition to the system firmware (also known as a BIOS), ProLiant systems contain several devices that have their own upgradable firmware chips. HP will usually only support your hardware if are running the latest versions of firmware.
HP provides several upgrade mechanisms, some upgrades are available via all methods, others only a subset. Most of the methods are Microsoft Windows centric and even the Linux utility uses a custom update script. The rumor is that the reason for this is that HP requires a way to prompt for the user to accept an EULA.
The mechanisms are,
This is a bootable DOS floppy that runs an update program. To create, you download a Microsoft DOS/Windows program which creates the floppy. This requires a way to run a Microsoft DOS/Windows executable on a system with a 3.5" floppy drive and a working writable 1.44mb floppy. Making a bootable DOS floppy, for use on a single floppy would be difficult since the the system would need to access both the program and the drive it's writing to at the same time. The program also only allows the option of writing to A: or B: drives (so you can easily direct it at a virtual drive) and requires a dos high memory manager to run.
In some cases users have reported success using Wine to create the floppy or, in the case of the iLO, extract the firmware for use by another method.
The Offline ROM Flash for SmartStart Maintenance is a package that allows system administrators to upgrade firmware on a server that's booted from the HP SmartStart CD, using the ROM Update Utility (available under the "Maintenance" tab). This requires a Smart Start CD, a system with a 3.5" floppy drive that can write an MS-DOS floppy (Linux or Microsoft Windows for example), and a working writable 1.44mb floppy.
This is a bootable Smart Start CD that contains a collection of firmware update images. This requires a paid "ProLiant Software Maintenance" subscription. It's very nice if you have it, it can update all firmware in one boot of the CD.
This is a Linux executable (wrapped in a shell script) that contains a Linux version of the firmware update software and the firmware image. It is run as root while booted Linux on the system to update. Some update utilites require that HP's system management software is installed in order to be able to access the device for the upgrade. This is not always possible in Debian, more on that below.
Of the above methods, the "Online ROM Flash Component for Linux" is the easiest and most accessible. In some cases, it's not possible to use that mechanism and an alternative way is available. This guide strives to recommend the easiest method possible while minimizing the use of propriatary software to do so.
HP provides an update tool for the system bios and several of the onboard chips all rolled in one. The update is platform dependent, one per unique system. For example: my dl360g3 system is also known as P31, different models and different generations of the same model will have different platform names.
Unfortunately HP's web pages are designed in such a way that there is no wayt to link to the latest version for a particular platform. Instead, traverse HP's support pages, as described above, until you find the 'Online ROM Flash Component for Linux' for your platform, a 'scexe' file. As described above this file is a shell script wrapper around a binary blob, which contains the update tool and firmware. On the systems I've tried (dl360g3) this utility runs fine under Debian. If you just use the --unpack option and look at what it provides, the update tool is a statically linked binary and it appears to just use general kernel methods to talk to the hardware, so I expect it will run fine on any Linux distribution. For example, on my system to update I did:
Downloaded the update file, 'CP005348.scexe'.
At a normal user prompt, ran '$ chmod +x CP005348.scexe'.
As root on the system to update, ran '# ./CP005348.scexe'.
Followed the instructions, waited to ensure it was finished, and then rebooted.
As with any firmware update, be very careful, mistakes can make your fancy expensive hardware into a dead piece of junk.
The iLO firmware is also made available through these same mechanisms, and you want the 'Online ROM Flash Component for Linux'. However if you attempt to run it you get 'ERROR: The HP Management drivers and agents are not installed.'. While it may be possible to get those installed (covered later in this document) there is an easier way. If you run the scexe with '--unpack=.' it will just unpack (in the current directory due to the '.') and one of the files it unpacks is the firmware image, named something like 'ilo182.bin'. You can use the ilo web interface and this file to upgrade the iLO firmware. For example in my case:
Downloaded the update file, 'CP005863.scexe'.
At a normal user prompt on my local system, ran '$ chmod +x CP005863.scexe'.
Ran it with the unpack flag '# ./CP005863.scexe --unpack=.'.
Pointed my web broswer at the iLO (see the iLO section of this document for info on using the iLO). Went to the Administration -> Upgrade iLO Firmware section. Clicked on the 'browse' button and selected the 'ilo182.bin' file. Then clicked 'Send firmware image' and waited to ensure the entire file was uploaded, then the browser redirected me to the iLO login page.
As noted in the firmware revision history, if you are upgrading from a version older than 1.60 (the first version where ssh was available) there is a one time delay while ssh keys are generated. So you may have to wait a while before you can access the ssh console.
The "integrated Lights Out" device is a special management processor (actually a mini-computer) integrated on most HP ProLiant servers that allows system monitoring, power control, console access, and other features. The iLO product documentation is available at http://h18004.www1.hp.com/products/servers/management/ilo/documentation.html. the "User Guide" and "Best Practices" documents provide a complete explaination of the iLO product. There is also a PCI card version called the "remote integrated Lights Out" (riLO).
This guide covers Debian issues for iLO that are not addressed in the documents above.
If your system is brand new it comes with a removable system information tag (attached via elastic or glue) that lists information such as:
Serial No: EAGZLDN42B User Name: Administrator DNS Name: ILOEAGZLDN42B Password: KVJJRUF4
login and password are case sensitive
If you don't have the iLO system information, see the troubleshooting selection below.
Once powered, the iLO sends out a dhcp request that includes the "DNS Name" from the system information tag in the request. Check your dhcp server log (/var/log/syslog on Debian) for the IP address your server issued. An example iLO request looks like:
dhcpd: DHCPDISCOVER from 00:0e:7f:b2:c1:2e via eth2 dhcpd: DHCPOFFER on 10.100.100.25 to 00:0e:7f:b2:c1:2e (ILOEAGZLDN42B) via eth2 dhcpd: DHCPREQUEST for 10.100.100.25 (10.0.0.1) from 00:0e:7f:b2:c1:2e (ILOEAGZLDN42B) via eth2 dhcpd: DHCPACK on 10.100.100.25 to 00:0e:7f:b2:c1:2e (ILOEAGZLDN42B) via eth2
Point your web browser at that IP address (10.100.100.25 in the above example) and record the MAC address for later use.
If using a non-Netscape/non-MSIE browser you may see a message that says HTTP 1.1 is not supported. Explicitly enter "https://" for the URL to work around this bug. (See the note in the Troubleshooting section)
Once connected to the iLO secure web page, use the login listed on the system information tag (or a login you setup via the RBSU) to login. Then complete the following steps in the web interface.
Administration -> User Administration
Change the Administrator passwd, setup additional accounts as needed. Note that "User Name" and "Login Name" can be confusing given the contradiction with the "User Name" on the system's removable tag. Don't confuse them or you can lock yourself out. ("User Name:" on the tag is actually "Login Name:" using the UI terminology)
If you do lock yourself out, you can use the ROM based Setup Utility (RBSU) to edit the accounts and fix the problem. See the RBSU section below for instructions.
The iLO has several ways of accessing the system console remotely, a telnet text console, a web java-based text console, and a web java-based graphics console. The telnet console is the easiest to setup and use and doesn't require a JVM on the client machine. Setting up the Java interface is left as an exercise for the user.
In the defalt iLO configuration the telnet console is not configured. First go to
Administration -> Global Settings
and set "Remote Console Port Configuration" to "Enabled" to turn on the telnet console. By default "Remote Console Data Encryption" is set to "Yes" meaning you need to use SSL enabled telnet. According to the iLO user manual SSL enabled telnet console is "encrypted with a 128 bit RC4 bi-directional cipher". The telnet-ssl command in Debian is able to connect with this setting enabled, however you can't type anything. The telnet-ssl(1) man page says ssl negotiation is supported, but encryption is not at this time. So set this option to "No" and ensure that the network that you use for the iLO is a secure non-public network (which is a good idea anyway) so that there is no possibility of someone sniffing your password.
Depending on how you access the remote console, your environment may intercept certain keypresses. To be able to use those keypresses on the server console you'll need to setup some hotkeys that do invoke the right key presses for you. Here is a sample setup of some things that might be useful.
Remote Console -> Remote Console Hot Keys
Setup these hotkeys:
Ctrl-T -> Ctrl-Alt-Del Ctrl-U -> F8 Ctrl-V -> F10 Ctrl-W -> F3 Ctrl-X -> NONE (we might need it for nano later) Ctrl-Y -> F12
If you are currently connected to the iLO via a dynamic IP address that your DHCP server assigned you will probably want to change to a static address. You can either configure your DHCP server to hand out a static address to the iLO (note that you have the MAC address in the DHCP server logs like in the example above) or you can hard code it using the web Administration -> Network Settings interface or via the ROM Based Setup Utility (explained below). If you have problems accessing the iLO after the address change, see the note in the troubleshooting section below.
Take a look around the web interface and note features that you might find useful. Some features require purchasing a license.
Your iLO is now configured, proceed to the next section.
Now that the iLO is setup, you should be able to connect to the console by telnet'ing to the iLOs IP address. You will be prompted for a login/passwd (which you setup above). The telnet console works when the system is in VGA text mode, this includes the initial POST of the system, the BIOS configuration utility, any peripheral configuration utilities (like the SmartArray configuration). As long as the software you are using stays in VGA text mode you will be able to see and interact with it as well, including the boot-loader and operating system itself. When the system is off or in graphics mode the telnet console will indicate that.
In addition to the telnet console you can now use the various features of the iLO web interface to control power and other settings
The iLO also has an XML programming interface for automating tasks. There is documentation as well as sample code at the URL listed at the beginning of the iLO section of this guide. Contribution of Debian specific details on using it would be appreciated.
Newer versions of the iLO firmware supposedly allow for ssh console, power control via the remote console, and other features. Details would be appreciated on that as well.
Now that we've reviewed how to configure and use the iLO, let's consider the security concerns around this device. The iLO is a networked computer and it's software resides in firmware. It is recommended not to use it on a non-secure subnet for several reasons,
Periodically security problems are discovered in standard network services, including those provided by the iLO. When that happens HP will need to issue a firmware update, which may take some time to be released and some time before the administrator gets around to upgrading (assuming they're paying attention at all). This delay is time that the iLO is exposed to a potential break-in.
Some methods of accessing the iLO, such as the telnet console, are insecure and can be sniffed by others on the network between the client and iLO. In addition to the iLO login/passwd, if the console is used to login to the system, switch to root, connect to other systems, etc. those keystrokes are going over the unencrypted telnet session.
Any device on a non-secure network can be subject to port scanning and denial of service attacks. It would be nice to avoid the iLO being subject to such attacks.
So the alternative is to put the iLO on a private network behind a firewall of some kind. Because the times when you are most likely to use the iLO are when the host system is unaccessable, this requires at least one other system. You can plug the iLO directly into a private network interface on a second system using a network crossover cable, or you can connect both and possibly other systems/iLOs to a network switch. Then you would configure the devices to use one of the RFC1918 defined private networks. If you want to get fancy you can even setup a DNS zone for this network and then address the iLO by name rather than IP. A whole HOWTO document could be written about this strategy.
Once the iLO is safely behind a firewall on a private network, you need to be able to access it. Using the telnet console is easy, connect to the gateway machine and then telnet to the iLO. Web access it slightly harder, you need to create a tunnel for your web traffic. You can use ssh to create a tunnel from a port on the client system to the https port on the iLO, via the gateway system. For example if the name of my gateway system is foo.bar.com and the IP of the iLO on the private network is 192.168.1.2, then on the client system run
ssh -L 8443:192.168.1.2:443 foo.bar.com
Then point your web browser at https://localhost:8443 (note the s in https://).
In addition the the web interface, the iLO has a console setup utility for changing some settings that you can use in case of trouble or if you prefer. You are prompted to enter durning the boot sequence. If you are unable to access the iLO web interface you can use the RBSU to change the iLO's network and login settings in order to use the web interface again.
If you don't know the IP address of the iLO
If the iLO is configured to use DHCP: you can force the iLO to do a DHCP request by either unplugging and replugging the it's LAN cable, or by removing all power from the server and reconnecting. While doing one of these, watch your DHCP server logs or run a program such as arpwatch to see the DHCP request and determine the IP (and MAC address if needed).
If the iLO is not using DHCP: use the RBSU from the console to change the network and user settings.
If you don't know the login/passwd for the web interface: use the RBSU to create a new login/passwd. If you also configured the RBSU itself to prompt for a login, then you will need to use a switch on the mainboard that disables the RBSU login so you can edit accounts. This process is documented in your server's system manual and possibly on the instructions on the inside of the case lid. See the HP website for manuals.
When accessing the iLO web interface you may get an error that your browser "does not support http 1.1". You can work around this by explictly specifying the secure interface by using "https://" with the URL. (The "http://" just redirects to the https:// address anyway, so you're just skipping this step)
When accessing the iLO with a browser if you may get a message about "incorrect Message Authentication Code". This is a bug that can show up in a couple different situations,
When accessing the iLO with a browser and changing its IP address, and then trying to access it at the new address using the same browser
Sometimes removing and reconnecting power, and then trying to access the iLO at the same address
It is not known if this is an iLO bug or a broswer bug. If you encounter this, you can work around the problem by either restarting your browser or using a different browser (or different instance of the same browser). Hopefully this will be fixed in newer firmware.