Mission Critical Linux Open Source Software
Home    mclx.com    CVS    Mailing lists    Contact us
Crash
Overview

Download

General usage

Commands / Help menu

Mailing list

...
 
Projects
Kimberlite Clustering Technology

In Memory Core Dump

Crash Core Analysis Suite

Kernel Compatibility Patches

Linux Support for Marvell's GT-64260 System Controller

dio

...
 
General Usage

These are the current prerequisites:

  • At this point, IA32, Alpha, PowerPC and IA64 based kernels are supported. Other architectures may be addressed in the future.
  • One size fits all -- crash can be run on any Linux kernel version from 2.2.5-15 through 2.4.*.
  • Similar to gdb, the kernel must be built with the top-level kernel Makefile's CFLAGS definition changed from: 
      CFLAGS = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
    to: 
      CFLAGS = -Wall -Wstrict-prototypes -O2 -g
    In 2.4 kernels, change the following line: 
      CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing
    to: 
      CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -g -fno-strict-aliasing
  • The deletion of -fomit-frame-pointer is done for dependable stack traces.
  • Also, as is the case with gdb, the uncompressed vmlinux kernel file created in the top-level kernel build directory must be saved.
To build the tool from the downloaded file, perform the following steps: 
  # tar xvzmf crash.tar.gz
  # cd crash
  # make

If desired, the name of the tool can be changed by simply modifying the PROGRAM= definition at the top of the Makefile.

If the tool is run against a crash dumpfile, two arguments are required, the uncompressed kernel name and the core dumpfile name.

If run on a live system, only the kernel name is required, as /dev/mem will be used as the dumpfile. If the kernel file is stored in either the /boot, /, /boot/efi or any subdirectory in /usr/src, then no command line arguments are required -- the first kernel found that matches /proc/version will be used as the namelist.

For example, if the command name crash is kept intact in the Makefile, invoking it on a live system would look like this: 

  # crash
          KERNEL: /boot/vmlinux
      DUMPFILE: /dev/mem (live system)
          CPUS: 1
          DATE: Mon Jan 5 23:01:34 2009
        UPTIME: 10 days, 22:55:18
  LOAD AVERAGE: 0.08, 0.03, 0.01
         TASKS: 42
      NODENAME: ha2.mclinux.com
       RELEASE: 2.4.0-test1
       VERSION: #11 SMP Thu Nov 4 15:09:25 EST 2000
       MACHINE: i686  (500 MHz)
        MEMORY: 128 MB
           PID: 3621                                  
       COMMAND: "crash"
          TASK: c463c000  
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)

  crash> help


                
  crash version:          gdb version: 
  For help on any command above, enter "help <command>".
  For help on input options, enter "help input".
  For help on output options, enter "help output".
  For the most recent version: http://www.missioncriticallinux.com/download

  crash>
When run on a dumpfile, both the kernel namelist and dumpfile names must be entered on the command line. For dumpfiles generated by the Mission Critical Linux patch, enter: 
  # crash vmlinux.17 lcore.cr.17
For dumpfiles generated by the SGI LKCD patch, enter: 
  # crash vmlinux vmdump.0
The tool's environment is context-specific. On a live system, the default context is the command itself; on a dump the default context will be the task that panicked. The most commonly-used commands are:
set set a new task context by pid, task address, or cpu.
bt backtrace of the current context, or as specified with arguments.
p print the contents of a kernel variable.
rd read memory, which may be either kernel virtual, user virtual, or physical.
ps simple process listing.
log dump the kernel log_buf.
struct print the contents of a structure at a specified address.
foreach execute a command on all tasks, or those specified, in the system.

Detailed help concerning the use of each of the commands in the menu above may be displayed by entering help command, where command is one of those listed above. Note that many commands have multiple options so as to avoid the proliferation of command names.

Command Output

Command output may be piped to external commands or redirected to files. By default, crash command output is piped to /usr/bin/less -E -X along with a prompt line. This behavior can be turned off in two ways:

  1. Enter set scroll off during runtime.
  2. Enter set scroll off in a .crashrc file, which can be located either in the current directory or in your home directory.
Command output may be piped to an external command using standard command line pipe syntax. For example: 
  crash> log | grep MHz
Command output may be redirected to a file using standard command line syntax. For example: 
  crash> foreach bt > bt.all
Use double brackets to append the output to a pre-existing file: 
  crash> ps >> crash.data

The default output radix for gdb output and certain crash commands is decimal. This can be changed to hexadecimal by using the built-in alias hex. It can be reverted back to decimal by using the built-in alias dec.

Command Input

Interactive crash commands are gathered using the GNU readline library, which implements a command line history mechanism, and a command line editing mode.

The command line history consists of a numbered list of previously run commands, which can be viewed by entering "h" at any time. A previously run command can be re-executed in a number of manners:

  1. To re-execute the last command, enter "r" alone, or "!!".
  2. Enter "r" followed by the number identifying the desired command.
  3. Enter "r" followed by a uniquely-identifying set of characters from the beginning of the desired command string.
  4. Recycle back through the command history list by hitting the up-arrow key until the desired command is re-displayed, and then hit ENTER. If you go too far back, hit the down-arrow key.

The command line editing mode can be set to emulate either vi or emacs. The mode can be set in the following manners, listed in increasing order of precedence:

  1. The setting of the EDITOR environment variable.
  2. An entry in either in a .crashrc file, which can be located either in the current directory or in your home directory. The entry must be of the form "set vi" or "set emacs".
  3. By use of the "-e" command line option, as in "crash -e vi" or "crash -e emacs".

To edit a previously entered command:

  1. Recycle back through the command history list until the desired command is re-displayed.
  2. Edit the line using vi or emacs editing commands, as appropriate.
  3. Hit ENTER.
If desired, a set of crash commands may be entered into a regular file that can used as input, again using standard command line syntax: 
  crash> < inputfile
An input file may be also be run from the crash command line using the -i option: 
  # crash -i inputfile
Lastly, if a command is entered that is not recognized, it is checked against the kernel's list of variables, structure, union or typedef names, and if found, the command is passed to p, struct, union or whatis. That being the case, as long as a kernel variable/structure/union name is different than any of the current commands, the appropriate command above will be executed. If not, the command will be passed on to the built-in gdb command for execution.

The .crashrc file, which can be located either in the current directory or in your home directory, may contain any number of set or alias commands -- see the help pages on those two commands for details.

Command Extensions

The crash utility has been designed to facilitate the task of adding new commands. New commands may be permanently compiled into the crash executable, or dynamically added during runtime using shared object files.

To permanently add a new command to the crash executable's menu:

  1. For a command named "xxx", put a reference to cmd_xxx() in defs.h.
  2. Add cmd_xxx into the base_command_table[] array in global_data.c.
  3. Write cmd_xxx(), putting it in one of the appropriate files. Look at the other commands for guidance on getting symbolic data, reading memory, displaying data, etc...
  4. Recompile and run.

Note that while the initial compile of crash, which configures and compiles the gdb module, takes several minutes, subsequent re-compiles to do such things as add new commands or fix bugs just takes a few seconds.

Alternatively, you can create shared object library files consisting of command extensions that can be dynamically linked into the crash executable during runtime or during initialization. This will allow the the same shared object to be used with subsequent crash releases without having to re-merge the command's code into each new set of crash sources. The dynamically linked-in commands will automatically show up in the crash help menu. For details, enter "help extend" during runtime, or enter "crash -h extend" from the shell command line.

Use of this website and the materials within are subject to the Terms and Conditions - any other use is strictly prohibited