HPUX Supplement for Micah Altman's Unix Handouts

Coming from an IRIX environment HPUX should not be too difficult to understand. Most of my notes on my main Unix page are directly applicable.

For HP-UX adminstrators, in addition to my general Unix reading list on that page, I recommend

Poniatowski, Marty, 1997. HP-UX System Adminstration Handbook and Toolkit,NJ:Prentice Hall.

HPUX Tools

Many of the common troubleshooting and tuning tools (such as ps) are available under HPUX, but may differ somewhat from other Unix'es. Here are some HPUX-specific tools or HPUX-specific implementations of common tools to look for:

Configuration

sam - HP's all-around system management and configuration tool.
swinstall (etc.) - HP's all-around system software management tool. (Also use swlist -l products, swlist -l filesets -a state -a install_date to list software and configurations, swverify to check installations.)
model - displays HP's model names for the system
nettune - Change network parameters (like "systune" on svr4 systems, for network variables).
sysdef - shows configuration of kernel parameters.
dmesg - shows system startup messages and memory configuration.
ioscan [-f] - Hardware inventory.
/usr/lbin/sysadm/system_prep -s - Recreates the "system" file, in /stand, which shows changes to default kernel variable settings, as well as other kernel installation features.
lsdev/lssof - Device driver inventory/details.

Tuning & Troubleshooting

top - top shows some information (RSS) that ps does not, and vice-versa
sar - Sar shows most of what you would expect from other systems. Be warned however, that the statistics for such things as inod-tables actually refer to the occupancy of the cache, not the table itself. (This is not properly documented, but was revealed to me by HP support.) Also, better paging statistics can be found in vmstat, some more i/o info in iostat
swapinfo [-m]|[-at] - Shows config & use of the swap devices.
lanadmin/landiag/lanscan - Can give network interface stats, show ethernet addresses. Supplemental to netstat, and can reset the counters.
netfmt -v -f /var/adm/nettl.LOG00 - List detailed troubleshooting messages from the lan interface
nettl - HP's packet tracer.
NFS - HP supports nfsstat,showmount and rpcinfo for NFS troubleshooting and tuning
DNS - Supports the d2 option for nslookup for performance and troubleshooting and logging for named using signals or sig_named. Also remember that in addition to /etc/resolv.conf (use "man resolver" to get information) HPUX uses /etc/nsswitch.conf (use "man switch" to get information).
sendmail- The sendmail options for resolving addresses are sendmail -bv -v -oL10 address (note that -bv prebvents message sending). Use sendmail -bt to get information about address rewriting. Use sendmail -v for mail tracing (to the next delivery agent).
what- this is a general utility for printing SCCS strings embedded in executables. It's particularly useful on HP's because they compile these into their code, and it an indicate which libraries were linked with the executable.
chatr - can be used to show libraries linked by an executable (Don't forget the old reliable tools for listing symbols and strings: nm and strings.)

Third-Party Tools

These free tools are available from software.hp.com, as well as other locations.

dmon - a curses-based performance monitor, makes the normal stats easier to monitor interactively.
lsof - like fuser, shows the files used by processes. Also shows other resource use, such as network ports. Invaluable for troubleshooting.
trace - Trace the system calls for an executable. Invaluable for troubleshooting. (Does not trace shell scripts, but remember that shell scripts can be traced by putting debug flags in the #! line.)
tcpdump - Trace packets on the network. Invaluable for network troubleshooting, can also be used as the foundation of a performance measurement tool.

HPUX Memory Management

HP-UX also has a significantly different approach to memory management than most other Unixes. Two kernel features may be of particular interest:

Dynamic Buffer Cache: This controls the amount of filesystem data cached in memory. Unlike other systems, DBC memory is not flushed when process space gets tight, so too large a dbc can cause swapping. Also, the DBC tends to be synced all at once, instead of paged gradually, so that large DBC's can cause intermittent delays. Tune with the kernel variables dbc_min_pct and dbc_max_pct.
Pseudo-Swap: HPUX does not have virtual swap, and in fact demands that there be swap allocation for every page of VM allocation, even if the VM currently resides in RAM. In other words, while IRIX allows safe VM allocation up to (RAM+SWAP), HPUX only allows it up to (SWAP). To get around this somewhat, HP uses a "pseudo-swap", and memory locking (which avoids the need for swap reservation) to avoid their own poor implementation of swap reservation. Sigh. This can cause problems on systems with a small amount of memory. Tune with swapmem_on and unlockable_pages.

You may also want to read the

HP-UX Kernel Tuning and Performance Guide .

HPUX Bootup Process

The HPUX bootup process is conceptually the same as the IRIX bootup process described in my handouts. Both OS's: (1) Load bootup parameters from non-volatile memory, and run diagnostics. (2) Load a bootstrap loader for the OS. (3) Start init and procede through the inittab entries and the rcXX.d scripts.

Some of the HPUX commands, especially in stages 1 & 2, are different in the details:

PDC: This is HP's firmware loader. To access it, press <esc> at the start of the boot process. The PDC menu will allow you to turn on the autoboot and autosearch parameters, which are useful if you want the system to come up automatically. The PDC menu also allows you to turn on diagnostics which can tell you if boards or memory is failing.
ISL: This is the second stage loader. To access it, use the "boot pri isl" command from PDC. From here, you can examine the boot path and console path, and also boot into single user mode, or boot a backup kernel using HPUX.
HPUX: The third (and final) stage loader. Use this from ISL to boot into single user mode or to boot an alternate kernel -- very useful when the system hangs on normal bootup (e.g. trying to access nonexistent NFS mounts). For example, to boot into single user mode, type "hpux -i1 boot" from ISL.
HP-ux proper: From here on, the boot process is quite similar to IRIX. Both use the inittab to direct the startup process. Most processes are started from the rc scripts (one big /etc/rc script in HPUX 9.X, and /sbin/rc?.d/* scripts in HPUX 10.0 and later). Many of the configuration parameters for these scripts are kept in /etc/rc.config.d. HPUX does not have a "verbose" startup mode, like IRIX, but instead keep a log of all boot script output in /etc/rc.log.

HPUX Panic Analysis

HP-UX does not support the svr5r3 tool icrash, which is immensely useful for crash debugging. Instead, they provide their own scripts. This is what you need to do to run a panic analysis:

cd /var/adm/crash/core.[n]
zcat /usr/contrib/lib/Q4Lib.tar.Z | tar -xvf -
gunzip vmunix
/usr/contrib/bin/q4pxdb vmunix
cp q4lib/sample.q4rc.pl ~/.q4rc.pl
/usr/contrib/bin/q4 -p .
include analyze.pl
run Analyze AU > crash[n]_rpt.txt

Search this site for: Search tips
[ Things to do with this page: | Print it! | Comment on it! | Track it! ]

Micah Altman