objdump parser

If you've been programming in Linux kernel, you must had the chance to debug kernel oops using objdump.
Normally, you invoke objdump with the name of the problematic driver, and then parse the output based on the information printed in the call trace of the kernel oops. Here is a nice example.


Among many hexadecimal numbers and function names that are printed in the call trace of the kernel oops, the most important piece of information is the function name and the offset: for example:
vnic_login_create_2+0xd1


I wrote this script that receives the function name and offset as a pointer, and prints the source file name and line number that caused the oops.


For example:
# objdump_parser.py vnic_login_create_2+0xd1 drivers/net/mlx4_vnic/mlx4_vnic.ko


vnic_login_create_2+0xd1 = 0x6e3 => drivers/net/mlx4_vnic/vnic_data_main.c:210

########################## objdump_parser.py ##########################

#!/usr/bin/env python
# -*- python -*-
# Author: Ali Ayoub

import os
import sys
import time

SNAME = os.path.basename(sys.argv[0])
USAGE = "%s " % (SNAME)
EXAMPLE = "%s %s %s" % (SNAME, "fip_vnic_login_create_1+0x4a", "mlx4_vnic.o")

# print functions
debug = "--debug" in sys.argv
if debug:
sys.argv.remove("--debug")

def vprint(msg):
if debug:
print "-V- %s" % msg

def eprint(msg):
print "-E- %s" % msg

def iprint(msg):
print "-I- %s" % msg

def main():
if len(sys.argv[1:]) != 2:
eprint("Bad Args")
iprint("USAGE: %s" % USAGE)
iprint("EXAMPLE: %s" % EXAMPLE)
return 1

trace = sys.argv[1]
trace_func = trace.split('+')[0]
trace_off = trace.split('+')[1]
mod = sys.argv[2]
objdump_f = "/tmp/%s.%s" % (os.path.basename(mod), time.time())

if (os.system("objdump -Sld %s > %s" % (mod, objdump_f))):
eprint("objdump failed")
return 1
vprint("objdump output %s" % objdump_f)
objdump_fd = open(objdump_f)
objdump_lines = objdump_fd.readlines()
objdump_fd.seek(0)
objdump_out = objdump_fd.read()

addr = 0
index = 0
addr_index = 0
token = "<%s>:" % trace_func
vprint("looking for %s in %s" % (token, objdump_f))

for line in objdump_lines:
index = index + 1
if not line.count(token):
continue
addr_index = index
vprint("found func %s at line %s (%s+%s)" % (trace_func, line.strip(), objdump_f, addr_index))
_addr = line.split()[0]
addr = hex(int(_addr, 16) + int(trace_off, 16))
break

if addr == 0:
eprint("couldn't find func %s" % trace_func)
return 1

vprint("%s -> %s" % (trace, addr))

# look for assembly line
found_line_index = 0
for i in range(int(addr, 16) + 4, int(addr, 16) - 16, -1):
token = "%s:" % hex(i)
token = token.replace("0x", '')
vprint("looking for hex add %s" % token)
index = 0
for line in objdump_lines:
index = index + 1
if line.strip().startswith(token):
found_line_index = index
break
if found_line_index:
break

if not found_line_index:
eprint("couldn't find found_line_index: %s" % found_line_index)
return 2

# print result
found_start_index = addr_index - 1
code_line=0
for i in range(found_start_index, found_line_index):
#print objdump_lines[i].strip()
if objdump_lines[i].strip().count(".c:") or objdump_lines[i].strip().count(".h:"):
code_line = i
if code_line:
print ("%s = %s => %s" % (trace, addr, objdump_lines[code_line].strip()))
# cleanup
objdump_fd.close()
os.remove(objdump_f)

return 0

if __name__ == '__main__':
try:
rc = main()
except Exception, e:
print "-E- Exception %s" % str(e)
print "-E- Abort."
sys.exit(rc)

"always on top" option for Windows

Always On Top Maker is a tiny interface less tool to make any window "always on top", or, make a topmost windows not always on top. How to use: Run AlwaysOnTopMaker.exe, then you can use 2 hot-keys, one is CTRL-ALT-T, is to switch the foreground application "always on top"/"not always on top", the other hotkey is CTRL-ALT-Q, is to exit Always On Top Maker.


Reference:
http://softzilla.blogspot.com/2009/06/always-on-top-windows-tools-best-always.html

netconsole for redhat

Net-Console module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical. It can be used either built-in or as a module. As a built-in, netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. While this doesn't allow capture of early kernel panics, it does capture most of the boot process. it's very useful for kernel developers to dump kernel oops callstack (source).

The configuration in Red Hat is very simple:

On the monitor machine:
1. Edit /etc/sysconfig/syslog and include the line SYSLOGD_OPTIONS="-m 0 -r -x"
2. Run dmesg -n 8
3. Restart syslog service: /etc/init.d/syslog restart

On the development machine (that may have kernel oops):
1. Edit /etc/sysconfig/netconsole and include the line SYSLOGADDR=(monitor-machine-IP)
2. Restart netconsole service: /etc/init.d/netconsole restart

Now you can work on the development machine and watch its kernel messages on the monitor machine, I watch the development machine by running this command on the monitoring machine:

#tail -f /var/log/messages | grep (devel-machine-ip)



Below some useful bash scripts for your convenience:

###################### netconsole_devel_set.sh ########################
#!/bin/bash
# Authour: Ali Ayoub -- ali@mellanox.com July 2010

USAGE="Usage: $0 "
EXAMPLE="Example: $0 <10.20.1.20>"

if [ $# != 1 ]; then
echo $USAGE
echo $EXAMPLE
exit 1
fi


ip=$1

egrep -v '^$|^#|SYSLOGADDR=' /etc/sysconfig/netconsole > /etc/sysconfig/netconsole.new
echo "SYSLOGADDR=$ip" >> /etc/sysconfig/netconsole.new
\cp /etc/sysconfig/netconsole /etc/sysconfig/netconsole.orig
\mv /etc/sysconfig/netconsole.new /etc/sysconfig/netconsole
dmesg -n 8
chkconfig netconsole on
/etc/init.d/netconsole restart
rc=$?
echo "-I- Done [$rc]."
exit $rc


###################### netconsole_monitor_set.sh ######################
#!/bin/bash
# Authour: Ali Ayoub -- ali@mellanox.com July 2010
if ( ps -lefc | grep syslog | grep syslogd | grep '\-m 0 \-r \-x' -q); then
echo "-I- Already set"
exit 0
fi

egrep -v '^$|^#|SYSLOGD_OPTIONS=' /etc/sysconfig/syslog > /etc/sysconfig/syslog.new
echo 'SYSLOGD_OPTIONS="-m 0 -r -x"' >> /etc/sysconfig/syslog.new
\cp /etc/sysconfig/syslog /etc/sysconfig/syslog.orig
\mv /etc/sysconfig/syslog.new /etc/sysconfig/syslog
chkconfig syslog on
/etc/init.d/syslog restart
rc=$?
echo "-I- Done [$rc]."
exit $rc

###################### netconsole_monitor.sh ##########################
#!/bin/bash
# Authour: Ali Ayoub -- ali@mellanox.com July 2010

USAGE="Usage: $0 "
EXAMPLE="Example: $0 <10.20.1.21>"

if [ $# != 1 ]; then
echo $USAGE
echo $EXAMPLE
exit 1
fi


ip=$1
cmd="tail -f /var/log/messages | grep ' $ip '"

if ! ( ps -lefc | grep syslog | grep syslogd | grep '\-m 0 \-r \-x' -q); then
echo "-E- please set SYSLOGD_OPTIONS='-m 0 -r -x'"
echo " in /etc/sysconfig/syslog and restart syslog service"
exit 1
fi
dmesg -n 8
nohup xterm -bg black -fg white -title "netconosle $ip" -hold -e "$cmd" &

SMS notification for Linux commands

Sometimes you need to run a Linux command that may take time (such as kernel compilation, virus scan, long test run..) instead of waiting in front of the monitor, you can do other stuff (e.g., good coffee) and you'll be notified when the command had finished with the results.

I have ATT cell phone, they support SMS2EMAIL service; simply send a message (plain text, no longer than 160 characters) to the address:

(your-number)@txt.att.net

For example:

4081002000@txt.att.net.

The Email address varies based on your carrier, check this website for worldwide cellular network list.

Now, we just need to run the command that takes time, and send a short text message with the report to notify us.. from a Linux machine with Email access, run this command:

(msg-body) | mail -s (msg-subject) (sms2email-address)

The message body can be the command output, or the command returned code, or anything else you want.

Examples:
a) To report the returned code of an application, run:
gcc windows.c; echo "rc=$?" | mail -s "gcc report" 4081002000@txt.att.net
This will send you the returned code (rc) of the compilation command when its done, usually zero (rc=0) indicates a successful run.


b) To compile a Linux kernel and send an SMS when done with the output tail (last 120 bytes for example), run:

make 2>&1 | tail -c120 | mail -s "Kernel Compilation Status" 4081002000@txt.att.net

Notes:
- You may need to adjust 'tail -c' flag to fit into 160 characters, see 'man tail' for more info.
- To redirect command error messages, use the suitable bash redirection methods.
- ATT sucks.

Fix Slow SSH Login Time

If your login times are really high, it may be that reverse DNS is not working correctly.

Solutions:

(a) Disable DNS in SSH daemon: go to /etc/ssh/sshd_config and include the line UseDNS no, then restart ssh service: /etc/init.d/sshd restart

(b) Or, fix your DNS configuration: in redhat, go to setup -> Network Configuration -> Edit DNS configuration -> and set the primary DNS and the search domain

Tips for better cscoping

Cscope is a developer's tool for browsing source code. It has an impeccable Unix pedigree, having been originally developed at Bell Labs back in the days of the PDP-11. Cscope was part of the official AT&T Unix distribution for many years, and has been used to manage projects involving 20 million lines of code! I use it a lot of work to manage browse the code of several kernels (any other editor I tried failed to do the job in reasonable time, cscope did handle it!)

Here are few tips for better cscoping:
- Display more pathname components in search results with -pN. By default, cscope only displays the basename of a given matching file. In large codebases, files in different parts of the source tree can often have the same name (consider main.c), which makes for confusing search results. By passing the -pN option to cscope at startup (or including -pN in the CSCOPEOPTIONS environment variable) -- where N is the number of pathname components to display -- this confusion can be eliminated. I've generally found -p4 to be a good middle-ground. Note that -p0 will cause pathnames to be omitted entirely from search results, which can also be useful for certain specialized queries.

- Use the built-in history mechanisms. You can quickly restore previous search queries by using ^b (control-b); ^f will move forward through the history. This feature is especially useful when performing depth-first exploration of a given function hierarchy. You can also use ^a to replay the most recent search pattern (e.g., in a different search field), and the > and < commands to save and restore the results of a given search. Thus, you could save search results prior to refining it using ^ (as per the previous tip) and restore them later, or restore results from a past cscope session.


Source:
http://cscope.sourceforge.net
http://blogs.sun.com/meem/entry/five_tt_cscope_tt_tips

Kernel Debuging Best Tool!

Finally! kernel debugging best tool has been revealed

Dual Monitor Background

I got recently two wide screen monitors for work, and I was looking for a way to set a background that fits both of them, this application does the job. It's very easy to use and for free.
This is how the desktop looks like with deep-sky background.