Search Ali's Blog

Saturday, July 10, 2010

netconsole for redhat


Net-Console module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical. It can be used either built-in or as a module. As a built-in, netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. While this doesn't allow capture of early kernel panics, it does capture most of the boot process. it's very useful for kernel developers to dump kernel oops callstack (source).

The configuration in Red Hat is very simple:

On the monitor machine:
1. Edit /etc/sysconfig/syslog and include the line SYSLOGD_OPTIONS="-m 0 -r -x"
2. Run dmesg -n 8
3. Restart syslog service: /etc/init.d/syslog restart

On the development machine (that may have kernel oops):

1. Edit /etc/sysconfig/netconsole and include the line SYSLOGADDR=(monitor-machine-IP)
2. Restart netconsole service: /etc/init.d/netconsole restart

Now you can work on the development machine and watch its kernel messages on the monitor machine, I watch the development machine by running this command on the monitoring machine:

#tail -f /var/log/messages | grep (devel-machine-ip)



Below some useful bash scripts for your convenience:
netconsole_devel_set.sh
netconsole_monitor_set.sh
netconsole_monitor.sh

Tip:
If you're using remote console switch, you can check the console output by invoking the command:
# echo `date`: Hello from `whoami` > /dev/ttyS0

To test netconsole, you just need to trigger any kernel module that prints messages into /var/log/messages, for example:
# modprobe -qr tun; modprobe tun


If your running in Dom0 of Xen/KVM, the netconsole script get confused by the interface bridging, to fix this, you need to update /etc/init.d/netconsole:
First, find the corresponding peth interface of the netdev interface you want to use  for netconsole, normally if you're using eth0, then the corresponding inetrafce in Xen is peth0, then find the MAC address of the remote logger IP, for example if the remote sys logger is 10.20.1.20, find its MAC by running:
# ping 10.20.1.20 -c 1 ; arp -n | grep 10.20.1.20
And finally, include the following two lines in /etc/init.d/netconsole:
 DEV=peth0 #your peth name here
 SYSLOGMACADDR=00:30:48:7D:DE:E4 #your MAC here
At this point you can use the scripts above to enable netconsole.

No comments: