Thursday, January 2, 2014

Linux Kdump

Why Kdump
Kdump is utility which really helps in determine the cause of kernel crashes, system hangs, or system reboots.

About Kdump

Kdump is the Linux kernel crash-dump mechanism. Redhat recommends that to enable the Kdump feature. In the event of a system crash, Kdump creates a memory image (vmcore) that can help in determining the cause of the crash. Enabling Kdump requires to reserve a portion of system memory for exclusive use by Kdump. This memory is unavailable for other uses.

Kdump uses kexec to boot into a second kernel whenever the system crashes. kexec is a fast-boot mechanism which allows a Linux kernel to boot from inside the context of a kernel that is already running without passing through the bootloader stage. & the system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel.

We can use common commands, such as cp and scp, to copy the memory image to a dump file on the local disk, or across the network to a remote system.

Architectures Support:

Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64,and s390x architectures.

How to configure Kdump ?

Prerequisites

For dumping cores to a network target, access to a server over NFS or ssh is required.

Whether dumping locally or to a network target, a device or directory with enough free disk space is needed to hold the core. See the "Sizing Local Dump Targets" section below for more information on how much space will be needed.

For configuring kdump on a system running a Xen kernel, it is required to have a regular kernel of the same version as the running Xen kernel installed on the system. (If the system is 32-bit with more then 4GB of RAM, kernel-pae should be installed alongside kernel-xeninstead of kernel.)

Configuring kdump on the Command Line:

To configure the amount of memory that is reserved for the kdump kernel on x86, AMD64, and Intel 64 architectures, open the /boot/grub/grub.conf file as root and add the crashkernel=<size>M@16M parameter to the list of kernel options

Different way to Configuring the Target Type

Edit the /etc/kdump.conf  & make appropriate change as per the production environment.

To change the local directory in which the core dump is to be saved, remove the hash sign (“#”) from the beginning of the #path /var/crash line, and replace the value with a desired directory path. Optionally, if you wish to write the file to a different partition, follow the same procedure with the #ext3 /dev/sda4

To write the dump directly to a device, remove the hash sign (“#”) from the beginning of the #raw /dev/sda5 line, and replace the value with a desired device name. For example:

To store the dump to a remote machine using the NFS protocol, remove the hash sign (#) from the beginning of the #net my.server.com:/export/tmp line, and replace the value with a valid hostname and directory path. For example:
To store the dump to a remote machine using the SSH protocol, remove the hash sign (#) from the beginning of the #net user@my.server.com line, and replace the value with a valid username and hostname. For example:
Enabling the Service

To start the kdump daemon at boot time, type the following at a shell prompt as root:
This will enable the service for runlevels 2, 3, 4, and 5. Similarly, typing chkconfig kdump off will disable it for all runlevels. To start the service in the current session, use the following command as root:
Testing the Configuration

To test the configuration, reboot the system with kdump enabled, and as root, make sure that the service is running:


Then type the following commands at a shell prompt as root: 


This will force the Linux kernel to crash, and the YYYY-MM-DD-HH:MM/vmcore file will be copied to the location you have selected in the configuration (that is, to /var/crash/ by default).

Analyzing the Core Dump

To analyze the vmcore dump file, you must have the crash and kernel-debuginfo packages installed. To do so, type the following at a shell prompt as root:
  ~]# yum install --enablerepo=rhel-debuginfo crash kernel-debuginfo


To start the utility, type the command in the following form at a shell prompt:
crash /var/crash/timestamp/vmcore /usr/lib/debug/lib/modules/kernel/vmlinux

Example: Running the crash utility



Example:  Displaying the kernel stack trace



To display a status of processes in the system, type the ps command at the interactive prompt. You can use ps pid to display the status of the selected process.

Example : Displaying status of processes in the system


Example: Displaying information about open files of the current context

How do I upload a large dump file to Red Hat Support Services?
In some cases, it might be necessary to send a kernel crash dump file to Red Hat Global Support Services for analysis. However, the dump file can be very large, even after being filtered. Since files larger than 250 MB cannot be uploaded directly through the Red Hat Customer Portal when opening a new support case, an FT P server is provided by Red Hat for uploading large files.

The FTP server's address is dropbox.redhat.com and the files are to be uploaded in the /incom ing/ directory. Your FTP client needs to be set into passive mode; if your firewall does not allow this mode, you may use the origin- dropbox.redhat.com server using active mode.

Make sure that the uploaded files are compressed using a program such as gzip and properly and descriptively named. Using your support case number in the file name is recommended. After successfully uploading all necessary files, provide the engineer in charge of your support case with the exact file name and its SHA1 or MD5 checksum.