xCAT Management Processor HOWTO

Introduction
IBM xSeries Management Processor
APC Master Switch
APC Master Switch Plus
Baytech Power Switch
IPMI Ethernet (LAN)
IPMI Serial (EMP)
WOL (Wake-on-LAN)
support

The purpose of this document is to describe the options available for the remote hardware management of IBM and non-IBM servers. This document assumes familiarity with xCAT 1.1.7.1 or 1.2.0 and the Building a Linux HPC Cluster with xCAT Redbook.

Hardware Management

xCAT provides a framework for the following hardware management functions:

Remote Power (on/off/stat/boot/cycle)
Remote Hard Reset
Remote Soft Reset (ctrl-alt-del)
Remote Hardware Vitals/Health Check
Remote Hardware Inventory
Remote OS Console
Remote BIOS/POST Console
Remote Hardware Event Logs
Remote MAC Address Collection
Remote Boot Destiny Control
Remote Beacon
Remote GUI Console

A management processor may provide all of some of the aforementioned functions. Through this document the follow table will illustrate which functions are available within xCAT for each management processor technology.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console

The above table directly correlates to the etc/nodehm.tab table. The value below the header will be the value to be used in the etc/nodehm.tab table. Values in green are currently implemented by the management processor technology and are supported by xCAT (e.g. mp). Values in yellow are currently implemented by the management processor technology but are not currently support by xCAT (e.g. under development). Vales in red indicate that the management processor technology does to provide that functionality (e.g. N/A).

Management Processors

Management processors (a.k.a. service processors and management controllers) are embedded systems within a server to provide access to the hardware management capabilities of the server. Management capabilities usually include, but are not limited to:

Remote power control. This is most important.
Remote hard reset
Remote hardware inventory
Remote vitals/health check
Remote hardware event log management
Remote console
Proactive hardware-based alerts

Access to the management processor is can be available from:

The OS. Some solutions require a device driver (e.g. IBM's Ranger management processor) and others can access the management processor directly through the OS using for example I2C (e.g. IBM's Hawk management processor and LM Sensors).
A serial port. Many solutions including IBM's MP and Intel's BMC have options for serial port access. IBM implements a standard VT100 and/or ANSI terminal interface whereas Intel BMC EMP requires a publicly documented protocol.
Out-of-band Ethernet. Out-of-band indicates that access to the management processor does not require access through the OS. E.g. IBM's RSA and ASMA-based Management Processor Networks allow access to individual management processors utilizing TCP/IP over Ethernet. IPMI-based solutions use UDP over Ethernet.

Servers and resources that do not have dedicated management processors can utilize remote managed Ethernet-based or serial-based power distribution units to provide some level of remote power management.

The IBM xSeries Management Processor

Each IBM xSeries server contains a Management Processor (MP) that is usually a PowerPC (Ranger) or H8 (Hummingbird and Hawk) chip with an embedded OS.

Useless Fact: The H8 chip is the same used by MIT and Lego for the Lego Mindstorms Robotic Invention System.

The IBM management processor provides but is not limited to the following functions:

Power control (on/off/state)
Reset (hard/soft)
Inventory (serial/model/proc/memory/etc...)
Vitals (fan speed/temp/voltages)
Alerts (SNMP)
Event Logging

Access to this management processor is available the following ways:

Ethernet. An Advance Systems Management Adapter (ASMA), Remote Supervisory Adapter (RSA), Remote Supervisory Adapter 2 (RSA2), or Blade Center Management Module (BCMM) is required for this method.
Serial. Not all servers have this as an option.
Device driver or I2C bus.
Management Processor Network (MPN).

The focus of this document is to use Ethernet/MPN with an ASMA, RSA, RSA2, or BCMM. Serial would require double the number of terminal servers and not all servers support this option. The device driver is not very useful is the machine is down or is running an OS with no device driver support.

IBM midrange and high-end xSeries servers ending in a zero (e.g. x330, x340, x360, x440) and Netfinity servers > 4000 contain a PowerPC-based management processor (Ranger) with OS, Serial, RS485, and Ethernet access available. xCAT only supports OS, RS485, and Ethernet access to Ranger.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
mp	mp	mp(4)	mp	mp	(1)	mp(2)	mp	NA	(3)

Only the x330 supports OS text console redirection through the management processor. To access this you must use the BIOS/POST console method (rvid/wvid). It is recommend for all servers that the serial port be used for OS text console.
Most IBM xSeries servers support serial BIOS/POST console. It is recommended to use the serial O/S Console method to access the BIOS/POST console. xCAT does not support character-based BIOS/POST console through RSA adapters, this feature is only available if using the ASMA.
Management processor-based GUI console is not supported in the xCAT framework. xCAT currently uses VNC for GUI Console (gcons) after the OS boots.
Soft Reset is only a feature of the x330.

IBM midrange and high-end xSeries servers ending in a two (e.g. x342) contain a H8-based management processor (Hummingbird) that is very limited in functionality and is not directly supported by xCAT, however this management processor can be augmented with a dedicated RSA adapters per server to provide the same functionality as any Ranger-based machine. i.e. Each x342 will require an RSA adapter.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
mp	mp	NA	mp	mp	(1)	NA(2)	mp	NA	(3)

To access this you must telnet to the RSA. It is recommend for all servers that the serial port be used for OS text console.
Most IBM xSeries servers support serial BIOS/POST console. It is recommended to use the serial O/S Console method to access the BIOS/POST console. An x342 with an RSA can provide character-based BIOS/POST console, however xCAT does not support character-based BIOS/POST console through RSA adapters. To access this you must telnet to the RSA.
Management processor-based GUI console is not support in the xCAT framework. xCAT uses VNC to GUI Console (gcons) after the OS boots. An x342 with an RSA can provide access to the VGA GUI of any OS. Use any Java-enabled web browser to access this feature.

IBM midrange and high-end xSeries servers ending in a five (e.g. x335, x345) contain a H8-based management processor (Hawk) that functions very similar to the Ranger but lacks dedicated serial access. xCAT only supports RS485 and Ethernet access to Hawk.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
mp	mp	NA	mp	mp(4)	(1)	NA(2)	mp	NA(5)	NA(3)

Hawk management processors do not provide access to the OS Console. It is recommend for all servers that the serial port be used for OS text console.
Most IBM xSeries servers support serial BIOS/POST console. It is recommended to use the serial O/S Console method to access the BIOS/POST console. An x342 with an RSA can provide character-based BIOS/POST console, however xCAT does not support character-based BIOS/POST console through RSA adapters.
Management processor-based GUI console is not support in the xCAT framework. xCAT uses VNC to GUI Console (gcons) after the OS boots. Any Hawk augmented with an RSA can provide access to the VGA GUI of any OS. Use any Java-enabled web browser to access this feature.
Hawk only provides MP ROM version and serial number.
Hawk provides a beacon, however the interface is not available through HTTP.

IBM Blade Center blades contain a H8-based management processor. The BCMM is the only access to this management processor.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
mp	mp	NA	mp	mp	(1)	mp	mp	mp	mp

Hawk management processors do not provide access to the OS Console. Use B/P Console.

Sample etc/nodehm.tab file:

# nodehm.tab
#
# node hardware management
#
# power     = mp,baytech,emp,apc,apcp,ipmi,NA
# reset     = mp,apc,apcp,ipmi,NA
# cad       = mp,NA
# vitals    = mp,ipmi,NA
# inv       = mp,NA
# cons      = conserver,tty,rtel,NA
# bioscons  = rcons,mp,NA
# eventlogs = mp,ipmi,NA
# getmacs   = rcons,cisco3500,cisco2950,bcnet
# netboot   = pxe,eb,elilo,nbgrub,file:,NA
# eth0      = eepro100,pcnet32,e100,bcm5700,e1000
# gcons     = vnc,NA
# sb        = N,Y (serial bios)
# beacon    = ipmi,NA
# bootseq   = mp,NA
#
#node	power,reset,cad,vitals,inv,cons,bioscons,eventlogs,getmacs,netboot,eth0,gcons,sb,beacon,bootseq
#
node1	mp,mp,mp,mp,mp,NA,mp,mp,cisco3500,pxe,eepro100,vnc,N,NA,NA
node2	mp,mp,NA,mp,mp,conserver,rcons,mp,rcons,pxe,eepro100,vnc,Y,NA,NA
node3	mp,mp,NA,mp,mp,conserver,NA,mp,rcons,pxe,bcm5700,vnc,Y,NA,NA
blade1	mp,mp,NA,mp,mp,NA,mp,mp,bcnet,pxe,bcm5700,vnc,N,mp,mp

node1 is an x330 with no terminal server and uses an ASMA for access to the management processor. In this configuration xCAT can provide power, hard and soft reset, vitals, inventory, and eventlogs. Remote serial console and BIOS console is not an option since there are no terminal servers. Using BIOS console through the ASMA/Ranger OS console can also be obtained. With no serial access to the machine MAC address collection utilizes the MAC table stored in a Cisco 3548 switch.

node2 is an x330 with a terminal server and uses an RSA for access to the management processor. In this configuration xCAT can provide power, hard reset, vitals, inventory, and eventlogs. Soft reset is not available through the RSA. With this machine connected to a terminal server and with serial BIOS/POST enabled in the BIOS, all console functions include MAC address collection are available using conserver.

node3 is an x335 with a terminal server and uses an RSA for access to the management processor. In this configuration xCAT can provide power, hard reset, vitals, inventory, and eventlogs. Soft reset is not available through the RSA. With this machine connected to a terminal server and with serial BIOS/POST enabled in the BIOS, all console functions include MAC address collection are available using conserver.

blade1 is an BC blade with no terminal server and uses an BCMM for access to the management processor. In this configuration xCAT can provide power, hard reset, vitals, inventory, beacon, bootseq, and eventlogs. Soft reset is not available through the BCMM. Remove video (rvid) is used for remote console.

Management Processor Function Chart

	x330 ASMA Chained	x330 RSA Chained	x340 ASMA Chained	x340 RSA Chained	x342 RSA Integrated	x335, x345 RSA Chained	Blade Center
rpower	X	X	X	X	X	X	X
rreset	X	X	X	X	X	X	X
rcad	X
rvitals cputemp	X	X	X	X	X	X	X
rvitals disktemp	X	X	X	X	X	X	X
rvitals ambtemp	X	X			X	X	X
rvitals voltage	X	X	X	X	X	X	X
rvitals fanspeed	X	X	X	X	X	X	X
rvitals power	X	X	X	X	X	X	X
rvitals powertime	X	X	X	X	X
rvitals reboots	X	X	X	X	X
rvitals state	X	X	X	X	X
rinv pci	X						X
rinv config	X	X		X	X
rinv model	X	X		X	X		X
rinv serial	X	X		X	X	X	X
rinv assett	X	X		X	X
rinv bios	X	X		X	X		X
rinv mprom	X	X		X	X	X	X
reventlog	X	X	X	X	X	X	X
rvid	X		X				X
rbootseq							X
rbeacon							X
serial bios	X	X			X	X

Management Processor Network

The IBM xSeries MPN is a RS485-based CAT5 or C2T cabled daisy-chained proprietary network of MPs (Management Processors). Access to any single Management Processor (MP) will allow access to any MP on the chain. Using this method it is only required to use one ASMA/RSA/BCMM per chain.

Illustration by Scott Denham, IBM

NOTE: Rangers and Hawks cannot be chained together. Technically it is supported by the hardware, xCAT does not support it.

Up to 12 Rangers (e.g. x330) may be chained together. NOTE: the ASMA/RSA is also a RS485-based device and will use up one of the 12 allowable devices on the chain. 8 or 10 nodes with 1 or 2 adapters is recommended. Rangers can be access by RSAs or ASMAs.

Up to 25 Hawks (e.g. x335) may be chained together. NOTE: the RSA is also a RS485-based device and will use up one of the 25 allowable devices on the chain. 10 or 16 nodes with 1 or 2 adapters is recommended. Hawks can only be accessed by RSAs. NOTE: the x335 does not have a dedicated RS-485 cable, but the chaining is done through the same cable as the KVM chaining.

Up to 14 blades share a single BCMM. The RS-485 cabling is internal to the Blade Center chassis.

Management Processor Adapters

As illustrated above a Management Processor Adapter is required to create a gateway from the MPN to Ethernet--xCAT only supports access to MPs through a MPA. Adapters are available in four different flavors, ASMA, RSA, RSA2, and BCMM. ASMA is 4 years old and has been replaced with RSA/RSA2. Extensive documentation on xSeries Management Processors and Management Adapters is available from http://www.ibm.com/support.

NOTE: ASMAs and RSAs are PCI adapters, but do not require any device drivers or OS support, and are externally powered. External power MUST be supplied. PCI adapters allow implementing a 0 U solution.

NOTE: BCMM is a module inserted in a dedicated management slot in the rear of a Blade Center chassis.

Each adapter supports three protocols:

Native, this is a proprietary protocol developed by IBM.
Telnet. (ASMA and RSA only).
HTTP.

For ASMAs, xCAT can be configured to automate the telnet or native protocol. For RSAs, only the native protocol is supported with Ranger-based (x330) chains, and only HTTP is supported with Hawks-based (x335) chains. Native protocol support is provide by the mpcli utility obtained from: http://www.ibm.com/support. xCAT includes a tested version of this tool. For BCMM only HTTP is supported.

Each ASMA, RSA, and BCMM must be assigned an IP Address and Subnet Mask and have defined /etc/hosts and DNS entries. For ASMAs you must use the ASMA boot diskette to assign IP and Subnet. For RSAs and BCMMs note the MAC address and use the xCAT static DHCP support to assign this or manually assign. Each ASMA, RSA, and BCMM must be listed as nodes in nodelist.tab.

To setup the ASMA/RSA for xCAT after the IP has been assigned type: mpasetup asma or rsa hostname

To setup the BCMM manually use the web interface and setup SNMP, alerts, and IP settings.

MPA/MPN commands:

mpasetup. This will setup the ASMA/RSA for alerting and other settings required for xCAT. BCMM not supported.
mpareset. Reboot the adapter.
mpascan. Scan for RS485 chained nodes.
mpacheck. Display the ASMA/RSA IP and Ethernet settings. Sanity check.
mpncheck. Verify that the MPN matchs the mp.tab table. Sanity check. BCMM not supported.

Defining your MPs and MPNs

To communicate with an MP to perform any function (e.g. power on) you must first define what MP is on what MPA. This is done in the mp.tab table. Another table to manage is the mpa.tab table. This table defines the characteristics of the MPA (e.g. ASMA or RSA, use telnet or native protocols, etc...).

Assuming that you successfully defined your cluster to xCAT and ran each node through stage1, stage2, and stage3, your MP names should equal the cluster hostnames (except BCMM). This name is used to navigate the MPN to select the correct node to perform any MP-based action (e.g. power off).

Use mpascan to verify that your MPs are on the correct MPA then define mp.tab. mp.tab is formatted:

OS Hostname<tab>MPA Hostname,Internal MP Name or Blade Bay Number

E.g.:

node1	asma1,node1
node2	asma1,node2
node3	asma1,node3
node4	asma1,node4
node5	asma1,node5
node6	rsa1,NA
node10	rsa1,node10
blade1	bc1,1
blade2	bc1,2

In this example, node1-node5 are managed though an MPA with the hostname of asma1, with node1-node5 as the corresponding internal MP name used on the MPN. It is strongly recommend that they match, if not SNMP alerts will have cryptic names. (xCAT's stage3 process takes care of this for you.) node6 and node10 are managed though an MPA with the hostname of rsa1. Notice that node6 has NA assigned as the internal MP name. This is a special case, some xSeries machines do not have on-board MPs or full function MPs, so an MPA is required per node (e.g. the x342). This adapter functions as the primary MP for that node but can also act as a MPA gateway to a MPN, notice that node10 is chained to this adapter. NA (capital NA) must be the Internal MP Name in the mp.tab table for xCAT to function properly. blade1-blade2 are on BCMM bc1 in bays 1 and 2.

NOTE: Hawk-based machines (e.g. x335) require that the RSA replace the onboard management processor similar to the above example with the x342. However unlike the x342 you can chain.

Use the mpncheck utility to verify that your MPNs are defined correctly. (Except BCMM)

Defining your MPAs

Because ASMAs and RSAs are not identical and function differently it is necessary to define how you want each MPA to function. This is defined in mpa.tab and is formated:

MPA hostname<tab>type,name,number,command,reset,rvid,dhcp,gateway,dns

MPA hostname, is the hostname you assigned your ASMAs, RSAs, and BCMMs in /etc/hosts and DNS. Should match the entries in mp.tab.
type, is the type of MPA (asma, rsa, or bc)
name is the internal name of the adapter on the MPN. Usually it is the same as the MPA hostname with one very important exception--MPAs that are the primary MP for that node. In this case the name would equal the OS hostname of that node, so that alerts have the correct information. This field is used by mpasetup. Unused by BCMM.
number, is an internal MPN index and must be unique, the xCAT convention is that they are number sequentially starting with 10001. This field is used by mpasetup. Unused by BCMM.
command, defines what protocol to use to issue commands to MPs. For type asma, both telnet and mpcli (native) are supported. mpcli is recommended. For type rsa, only mpcli is supported for chained Rangers (e.g. x330), and only http is support for chained Hawks (e.g. x335). For type bc, only http is supported.
reset, defines what protocol to use to issue a MPA reboot. For type asma, both http and mpcli (native) are supported. http is recommended. For type rsa, mpcli and http is supported. It is strongly recommend that the reset method not equal the command method. If the command method is unavailable for any reason, a different protocol should be use to reset the adapter. For ASMA use a telnet,http|mpcli or mpcli,http pair. For RSA use mpcli,http for Ranger chains, and http,mpcli for Hawk chains. For type bc, only http is supported.
rvid, defines what protocol to use for remote video access. Only telnet is supported and only for type asma. The mpcli tool that xCAT uses to provide native protocol support for MPAs does not support remote video access (yet). It is recommended that remote video be access through the COM1: serial port of the node. For type bc, only http is supported.
dhcp, is used to assign the MPA IP address. If set to Y, then DHCP will be used, if set to N, then the IP address of the MPA as assigned in DNS will be used. ASMA adapters do not support DHCP. The manual initial assignment of IP addresses to both RSA and ASMA is still required, however once set mpasetup can be used to change/update the assignment usually from DHCP to static for RSA adapters.
gateway, is the default IP gateway. If no gateway is required or if dhcp is set to Y, then NA can be assigned.
dns, is the default DNS server. If using dhcp set to NA. This field is required for RSA's managing Hawk-based management processors.

E.g.:

asma1	asma,asma1,10001,telnet,http,telnet,N,199.88.179.1,NA
rsa1	rsa,rsa1,10002,mpcli,http,NA,Y,NA,NA
rsa2	rsa,node20,10003,http,mpcli,NA,N,199.88.179.1,199.88.179.20
bc1	bc,bc1,NA,http,http,http,N,NA,NA

Remote Video

xCAT provides two paths to the remote video VGA character screen.

Through an MP from an MPA.
Through the COM1:/COM2: on the node using Serial Console.
Through the BCMM using VNC.

Because of the nature of MPAs/MPNs only one node per MPN/MPA can communicate. For short commands (e.g. power stat) this does not pose a problem, but for long sessions like remote video access the MPN chain and MPA can be tied up exclusively for a long period of time.

To use first method with ASMA, just use the rvid or wvid commands. With RSA you have to manually telnet to the adapter, login, chain, and then run the remote video--not as automated, but simple enough. I recommending typing:

TERM=vt100 telnet adapter;tput sgr0

NOTE: Do NOT use rvid/wvid Remote Video when flashing BIOSes remotely or manually. You will hose your machine. If you do hose a node refer to the HMM for your node type on BIOS recovery procedures. Use cerial Console to monitor BIOS flash.

The second method is only available for the x330, x342, x235, x335, and x345 with latest BIOS.

Obtain the latest BIOS, flash, enable Serial Console for 9600 8N1, VT100, save and reboot. The new stage1.dd/stage1.iso also includes the new BIOS.

For BCMM rvid using VNC and it performs very well for test and GUI, however only one session per BCMM.

The MPCLI tool

A new addition to xCAT is the IBM developed MPCLI tool. This java-based tool is used to communicate with the MPA using native protocols. Updates to this tool are available from http://www.ibm.com/support.

You must define the root directory of the MPCLI tool as mpcliroot in site.tab (see the samples/etc directory). To use the MPCLI included with xCAT this must be set to $XCATROOT/lib/mpcli. If xCAT was installed in a directory other than /opt, then use the correct path.

Sample entry:

mpcliroot $XCATROOT/lib/mpcli

Authentication

All IBM xSeries MP technology requires authentication. Until recently xCAT hardcoded the userid and password to use the defaults. It is now required to define the userid and password in passwd.tab (see the samples/etc directory).

Sample entries:

asmauser USERID
asmapass PASSW0RD

It is also required that the userids and passwords be consistent across the entire cluster. It is recommended to use the defaults. If you must use different a different global userid and password, then update the stage3.tmpl in xcat/stage and type ./mkstage. This will allow stage3 to set the userid and password for the on-board MP. To set the MPA userid and password you will have to use the MPCLI tool manually. Future versions of xCAT will assign the userid and password to both MPs and MPAs as part of stage3 an mpasetup.

If you require that each MP and MPA have unique userids and passwords or multiple userids and passwords per MP and MPA, then let me know and I will add it as a requirement to be released in a future version of xCAT.

The APC Master Switch

Extended description TBD.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
apc	apc(1)	NA	NA	NA	NA	NA	NA	NA	NA

Hard Reset is simulated with a power cycle.

The APC Master Switch Plus

Extended description TBD.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
apcp	apcp(1)	NA	NA	NA	NA	NA	NA	NA	NA

Hard Reset is simulated with a power cycle.

The Baytech Power Switch

Extended description TBD.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
baytech	NA	NA	NA	NA	NA	NA	NA	NA	NA

The Baseboard Management Controller

The Baseboard Management Controller (BMC) is native to IBM, Intel, and OEM system boards. Remote access to the BMC can be established with serial or LAN. For Serial use EMP and for LAN use IPMI.

IPMI (Ethernet/LAN)

The Baseboard Management Controller (BMC) IPMI LAN support allows the concurrent use of the on-board NIC as both an OS NIC and as a management NIC. Management traffic is diverted to the BMC before the OS can capture it. This LAN-based management solution also allows IPMI LAN-based commands to be sent with the system power off.

xCAT supports two different IPMI solutions (Each solution is detailed below):

xCAT's native support
Sun's ipmitool

NOTE: Before you can start to issue remote commands the BMC must be setup to accept remote commands via the LAN. Read the xCAT Stage1 HOWTO for details on how to setup the BMC for various systems.

NOTE: The xCAT stage3 code supports the e325 for automatic setup of the BMC. You must define a unique IP address for the BMC that differs from the OS IP for that node. Read below on the setup of ipmi.tab and passwd.tab. This must be setup before you run stage3.

IANS:

Enable the BMC for LAN traffic and assign a USERID and PASSWORD with administrator rights. Some systems may require that the NIC be enabled for pass-through support.
Forward SNMP alerts to your management server.
Assign an IP address. Some implementations support DHCP and static IP assignment, other static only. In either case you must have an IP assigned.

E.g.

If your IPMI solution (e.g. Intel anything, IBM x382, x343) uses the same MAC address for IPMI and eth0 and your BMC is setup for DHCP, then collect MACs using stage2 reboot and your are ready.

If either case if you wish to use static assignment (e.g. e325), then no DHCP setup is required.

Universal to both of the methods, the IPMI USERID and PASSWORD must be entered into $XCATROOT/etc/passwd.tab as ipmiuser and ipmipass. Also universal is the table $XCATROOT/etc/ipmi.tab. This table defines the xCAT nodename to IPMI IP address mapping. Often the IPMI IP address is not the same as the host address. If no entry exists for the node, then the node's host IP is used as defined in DNS.

Unique to the ipmi method the fields ipmimaxp, ipmitimeout, and ipmiretries must be defined in $XCATROOT/etc/site.tab.

ipmimaxp is the maximum number of concurrent IPMI sessions a single command (e.g. rpower) will have running, this prevents the management server from being overloaded with processes and increases the reliability of IPMI. Increase for scalability and decrease for stability. Recommended value: 100

ipmitimeout is the maximum number of seconds the command will wait for a successful results. xCAT will try over and over again up to the ipmitimeout. Increase for reliability, you must also increase if you increase ipmimaxp. Decrease for quicker timeouts. Recommended value: 3

ipmiretries is the maximum number of retries that will be attempted. ipmiretries * ipmitimeout = the maximum amount of time a single operation will be allow to take. Keep both ipmiretries and ipmitimeout small. Recommended value: 10

Unique to the ipmi method the optional field ipmisdrcache may be defined in $XCATROOT/etc/site.tab.

ipmisdrcache if set to yes will cache to /var/cache/xcat the contents of the BMC SDR. Reading the entire SDR takes time and the information is static. If caching is enabled the first read of the SDR will be cached for that node type and BMC version. Subsequent reads will take the SDR from local cache. The SDR is used by SNMP alert decoding, revenlog, and rvitals. Recommended value: yes

Unique to the ipmi method is syslog reporting. Check /var/log/messages for detailed error output and SNMP alerts.

xCAT's ipmi

xCAT's IPMI support is written in 100% Perl and is the native method.

Use this method for any IPMI-based system.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
ipmi	ipmi	NA	ipmi	ipmi	(1)	(1)	ipmi	ipmi	NA(2)

SOL support for e325 only. All others use physical serial port.

To enable e325 SOL follow the notes in the xCAT stage1 HOWTO. Then add a similar line to $XCATROOT/etc/conserver.cf:

nodename:|sol.e325 nodename::&:
BMC processor-based GUI console is not supported by IPMI. xCAT currently uses VNC for GUI Console (gcons) after the OS boots.

Sun's ipmitool

ipmitool is a C-based CLI interface to the LAN BMC. xCAT automates ipmitool by calling it directly.

ipmitool is available from http://ipmitool.sourceforge.net. This interface was written by Garrick Staples <garrick at usc.edu>.

Contact Garrick for a list of tested systems.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
ipmitool	ipmitool	ipmitool	ipmitool(1)	ipmitool(2)	(3)	(3)	ipmitool	NA	NA(4)

disktemp, powertime, reboots, and state are NOT supported.
model, serial, asset, and all are supported.
OS, BIOS, and POST console can be available through IPMI Serial-On-LAN (SOL), however some IPMI implementations completely monopolizes the IPMI LAN interface preventing other remote management functions. My recommendation is that the serial port be used for OS, BIOS, and POST console. Currently IPMI-based machines support serial-BIOS/POST and OS. NOTE: The e325 does NOT monopolize the LAN interface.
BMC processor-based GUI console is not supported by IPMI. xCAT currently uses VNC for GUI Console (gcons) after the OS boots.

IPMI (Serial/EMP)

Extended description TBD.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
emp	NA	NA	NA	NA	NA	NA	NA	NA	NA

WOL (Wake-on-LAN)

Extended description TBD.

Power	Hard Reset	Soft Reset	Vitals	Inventory	OS Console	B/P Console	Event Logs	Beacon	GUI Console
wol(1)	NA	NA	NA	NA	NA	NA	NA	NA	NA

WOL will only power on the machine. Where is Wake-off-LAN? Since Linux never crashes a simple ssh node halt will power off the node if properly configured.

Support

http://xcat.org

Egan Ford
egan@us.ibm.com
July 2004