Why Kdump
Kdump is utility which really helps in determine the cause of kernel crashes, system hangs, or system reboots.
About Kdump
Kdump is the Linux kernel crash-dump mechanism. Redhat recommends that to enable the Kdump feature. In the event of a system crash, Kdump creates a memory image (vmcore) that can help in determining the cause of the crash. Enabling Kdump requires to reserve a portion of system memory for exclusive use by Kdump. This memory is unavailable for other uses.
Kdump uses kexec to boot into a second kernel whenever the system crashes. kexec is a fast-boot mechanism which allows a Linux kernel to boot from inside the context of a kernel that is already running without passing through the bootloader stage. & the system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel.
We can use common commands, such as cp and scp, to copy the memory image to a dump file on the local disk, or across the network to a remote system.
Architectures Support:
Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64,and s390x architectures.
How to configure Kdump ?
Prerequisites
For dumping cores to a network target, access to a server over NFS or ssh is required.
Whether dumping locally or to a network target, a device or directory with enough free disk space is needed to hold the core. See the "Sizing Local Dump Targets" section below for more information on how much space will be needed.
For configuring kdump on a system running a Xen kernel, it is required to have a regular kernel of the same version as the running Xen kernel installed on the system. (If the system is 32-bit with more then 4GB of RAM, kernel-pae should be installed alongside kernel-xeninstead of kernel.)
Configuring kdump on the Command Line:
To configure the amount of memory that is reserved for the kdump kernel on x86, AMD64, and Intel 64 architectures, open the /boot/grub/grub.conf file as root and add the crashkernel=<size>M@16M parameter to the list of kernel options
Different way to Configuring the Target Type
Edit the /etc/kdump.conf & make appropriate change as per the production environment.
To change the local directory in which the core dump is to be saved, remove the hash sign (“#”) from the beginning of the #path /var/crash line, and replace the value with a desired directory path. Optionally, if you wish to write the file to a different partition, follow the same procedure with the #ext3 /dev/sda4
To write the dump directly to a device, remove the hash sign (“#”) from the beginning of the #raw /dev/sda5 line, and replace the value with a desired device name. For example:
Kdump is utility which really helps in determine the cause of kernel crashes, system hangs, or system reboots.
About Kdump
Kdump is the Linux kernel crash-dump mechanism. Redhat recommends that to enable the Kdump feature. In the event of a system crash, Kdump creates a memory image (vmcore) that can help in determining the cause of the crash. Enabling Kdump requires to reserve a portion of system memory for exclusive use by Kdump. This memory is unavailable for other uses.
Kdump uses kexec to boot into a second kernel whenever the system crashes. kexec is a fast-boot mechanism which allows a Linux kernel to boot from inside the context of a kernel that is already running without passing through the bootloader stage. & the system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel.
We can use common commands, such as cp and scp, to copy the memory image to a dump file on the local disk, or across the network to a remote system.
Architectures Support:
Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64,and s390x architectures.
How to configure Kdump ?
Prerequisites
For dumping cores to a network target, access to a server over NFS or ssh is required.
Whether dumping locally or to a network target, a device or directory with enough free disk space is needed to hold the core. See the "Sizing Local Dump Targets" section below for more information on how much space will be needed.
For configuring kdump on a system running a Xen kernel, it is required to have a regular kernel of the same version as the running Xen kernel installed on the system. (If the system is 32-bit with more then 4GB of RAM, kernel-pae should be installed alongside kernel-xeninstead of kernel.)
Configuring kdump on the Command Line:
To configure the amount of memory that is reserved for the kdump kernel on x86, AMD64, and Intel 64 architectures, open the /boot/grub/grub.conf file as root and add the crashkernel=<size>M@16M parameter to the list of kernel options
Edit the /etc/kdump.conf & make appropriate change as per the production environment.
To change the local directory in which the core dump is to be saved, remove the hash sign (“#”) from the beginning of the #path /var/crash line, and replace the value with a desired directory path. Optionally, if you wish to write the file to a different partition, follow the same procedure with the #ext3 /dev/sda4
To
store the dump to a remote machine using the NFS protocol, remove the
hash sign (“#”)
from the beginning of the #net
my.server.com:/export/tmp line,
and replace the value with a valid hostname and directory path. For
example:
To
store the dump to a remote machine using the SSH protocol, remove the
hash sign (“#”)
from the beginning of the #net
user@my.server.com line,
and replace the value with a valid username and hostname. For
example:
Enabling the Service
To start the kdump daemon at boot time, type the following at a shell prompt as root:
This will enable the service for
runlevels 2, 3, 4, and 5. Similarly, typing chkconfig kdump off will
disable it for all runlevels. To start the service in the current
session, use the following command as root:
Testing the Configuration
To test the configuration, reboot the system with kdump enabled, and as root, make sure that the service is running:
Then type the following commands at a shell prompt as root:
This will force the Linux kernel to crash, and the YYYY-MM-DD-HH:MM/vmcore file will be copied to the location you have selected in the configuration (that is, to /var/crash/ by default).
Analyzing the Core Dump
To analyze the vmcore dump file, you must have the crash and kernel-debuginfo packages installed. To do so, type the following at a shell prompt as root:
~]# yum install --enablerepo=rhel-debuginfo crash kernel-debuginfo
To start the utility, type the command in the following form at a shell prompt:
crash /var/crash/timestamp/vmcore /usr/lib/debug/lib/modules/kernel/vmlinux
Example: Running the crash utility
Example: Displaying the kernel stack trace
To display a status of processes in the system, type the ps command at the interactive prompt. You can use ps pid to display the status of the selected process.
Example : Displaying status of processes in the system
Example: Displaying information about open files of the current context
How do I upload a large dump file to Red Hat Support Services?
In some cases, it might be necessary to send a kernel crash dump file to Red Hat Global Support Services for analysis. However, the dump file can be very large, even after being filtered. Since files larger than 250 MB cannot be uploaded directly through the Red Hat Customer Portal when opening a new support case, an FT P server is provided by Red Hat for uploading large files.
The FTP server's address is dropbox.redhat.com and the files are to be uploaded in the /incom ing/ directory. Your FTP client needs to be set into passive mode; if your firewall does not allow this mode, you may use the origin- dropbox.redhat.com server using active mode.
Make sure that the uploaded files are compressed using a program such as gzip and properly and descriptively named. Using your support case number in the file name is recommended. After successfully uploading all necessary files, provide the engineer in charge of your support case with the exact file name and its SHA1 or MD5 checksum.
