Search Ali's Blog

Thursday, December 09, 2010

Xen Installation on CenOS-5.5

As part of my work, I had to bring up Xen Servers setup, and install some virtual machines.
I've been working on VMware ESX Servers for a while, and I must admit that VMware products are way more mature than Xen products. Below step by step installation of Xen over CentOS-5.5 and how to bring up and configure DomU Virtual Machines.

Dom0 Installation:
1. Install CentOS-5.5 OS
   ISO is downloadable from here

2. Install Xen kernel and Tools:
# yum install kernel-xen kernel-xen-devel xen

# yum install virt-manager
# yum install virt-viewer
3. Edit /etc/grub.conf and select xen kernel

title CentOS (2.6.18-194.26.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.26.1.el5
        module /vmlinuz-2.6.18-194.26.1.el5xen ro root=LABEL=/ rhgb quiet
        module /initrd-2.6.18-194.26.1.el5xen.img
title CentOS (2.6.18-194.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ rhgb quiet
        initrd /initrd-2.6.18-194.el5.img
4. Reboot the machine
5. Check kernel name, it will have xen substring:

# uname -a
Linux 2.6.18-194.26.1.el5xen ...
6. On my machine, yum installed Xen version 3.0.3:

# rpm -qa|grep -i ^xen-[0-9]

DomU Installation:
1. run virt-manager from dom0
2. select server name in the table, and select "new"

3. configure the virtual machine, when you get to the installation media, you better have the installation dir (extracted iso) available through on your network (check this link if you want to bring up your own apache) note that for full Linux installation (e.g., for RHEL-5.5) you need more than 4GB virtual disk space (I use 8GB).
4. when installation is done, machine will reboot and come up with bridged network connection
5. If you configured your VM to use DHCP, then you're ready, go ahead and ping the WAN.
- For more info on how to use virt-manager to install DomU instance (with snapshots) check this link.

DomU Network Configuration:
Xen supports three modes for network virtualization:
Bridging, Routing w/o NAT, Routing with NAT, each one of them need to be configured differently.

By default, the network is set up in bridging mode.
To change the network configuration, there are two main configuration files:
# /etc/xen/xend-config.sxp (hypervisor conf)

# /etc/xen/VM-NAME (virtual machine conf)

DomU in Bridging mode:
1. Make sure that xend-config.sxp has the following lines:
   (network-script network-bridge)

   (vif-script vif-bridge)
   To use different netdev (rather than eth0) use the line:
   (network-script 'network-bridge netdev=eth1')

   To create multiple bridges, you must create your own script.
   For example, in 
xend-config.sxp add:
   (network-script network-bridge-multi)
   The cerate the file (with +x permissions):   # cat /etc/xen/scripts/network-bridge-multi
   dir=$(dirname "$0")
   "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
   "$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1
   "$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2
   check this link for more details.
2. Configure DomU virtual interfaces,
   Edit /etc/xen/VM-NAME and examine the following line:

   vif = [ "mac=00:16:36:69:0d:0b,script=vif-bridge,bridge=xenbr2" ]
   where xenbr2 is an example of the bridge name
3. Note that if the netdev of the bridge your going to use was created after machine reboot, then you need to restart the xen network script, for example:
# /etc/xen/scripts/network-bridge stop 
# /etc/xen/scripts/network-bridge start
If the pethX interface you expect does not come up, try to reboot the machine (I noticed that in Xen 3.0, the network-bridge may fail to create the pethX interface if you restart the script few times, reboot solved the issue)
4. Make sure that the bridge as appear in brctl is output, is configured as expected:
#brctl show xenbr2
bridge name     bridge id  STP enabled  interfaces
xenbr2          8000.00..  no           vif2.0
If you configured your system in the past to work in routing mode, you may see virbr0 instead of xenbr0, to revert these changes, run:
# service libvirtd stop; chkconfig libvirtd off
If for some reason the peth2 that coresponds to your netdev is not attached to the bridge, then find your peth:
# ethtool -i peth2
and then add the interface to the bridge:
#  brctl addif xenbr2 peth2
If for some reason, peth2 link is down, run:
# ip link set peth2 up

At this point you can run the VM, configure the IP, and heck ping-pong, you can tcpdump the bridge (xenbr2) and its slaves to check the packets flow.

DomU in Routed mode:
Note that most of the tips of this section are based on this link.
1. Update your grub entry to include the following parameters:

title Kernel-XXX-xen
    root (hd0,5)
    kernel /boot/xen.gz 
    module /boot/vmlinuz-XXX-xen root=/dev/sda6 vga=0x31a resume=/dev/sda5 splash=silent showopts
    module /boot/initrd-XXX-xen
2. Set netloop driver module parameter, edit /etc/modprobe.conf and include the line (to top netloop from creating 8 useless vifs):
options netloop nloopbacks=0
3. Reboot Dom0
4. Configure Dom0 networking, edit /etc/xen/xend-config.sxp and include the line:

(network-script network-route)
(vif-script     vif-route)
5. Configure DomU networking, edit /etc/xen/VM-NAME and include the line:
vif = [ 'mac=00:16:3e:33:22:11, ip=, vifname=eth0' ]
Where the IP address, is the IP address you plan to give to vifname interface within the VM.
6. Power on the VM, and conifgure (for example) the eth0 interface to use the IP=
7. Ping the IP address of Dom0 (run ifconfig from Dom0 and check your netdev IP, must start with 192.x.x.x [depends on your mask]).
8. To ping other machines outside the GW, need to configure the default GW in the VM, this can be done in many different ways, for example:
route add default gw eth0

- To install KVM on CentOS, check this link

- If you get the error message:
ERROR    unable to connect to 'localhost:8000': Connection refused
You probably don't have xen-http-server enabled, edit:
and include the line:
(xend-http-server yes)
then restart xend:
/etc/init.d/xend restart

Thursday, December 02, 2010

Google Currency converter

Nice and simple Google Currency converter:
Example, from USD to ILS:

Tuesday, October 26, 2010

GIT Tips

While looking for GIT configuration for "signed-off-by" line, I found this nice link for GIT tips.
It has nice git-config commands that you might find useful in general.

Specifically, to automaticaly add the "signed-off-by: NAME " line into your git commits, run the following two commands:

git config 'Your name'
git config ''

I also found this nice link while searching for git cherry-pick, thanks for sharing.

Tuesday, October 05, 2010

Extract vSphere Installation Bundle

If you happen to need to extract vSphere Installation Bundle (vib), this can be done by running the following:
- Extract the zip archive:
Two files will be created, and VIB.vib
- Extract the vib archive
  ar -x VIB.vib
The following files will be created:
  short.rpm sig.pkcs7
- Extract data.tar.gz
  tar -xzvf data.tar.gz
  The result would be the files you are looking for. enjoy!

- ar command is normally not available on ESX Server, run this command on Linux.
- I added support to this script to recognize vib extension.

Thursday, September 30, 2010

Run as daemon in Linux

In Linux, when a command is executed from a terminal window, it will be child of that window's PID. For example, see below how xlogo is parent of the terminal:
# open terminal window:
# echo $$
# xlogo &
# ps -l | grep xlogo
  0 S 1049 4956 4830 0 75 0 - 9979 - pts/11 00:00:00 xlogo

As a result, whenever you close the terminal (parent) window, the xlogo will terminate.

If you wish to run your application as daemon, use the command NOHUP(1), this will make the running application immune to hangups. For example, see how xlogo is child of PID 1 (the init process):
# open terminal window:
# echo $$
# nohup xlogo &
# close the terminal window
# ps -l|grep xlogo
  0 S 1049 5013 1 0 75 0 - 9979 - pts/11 00:00:00 xlogo

Tuesday, September 28, 2010

SSH Host key verification failed

If you keep getting this annoying SSH failure:

I used to edit known_hosts file and remove the record that coresponds to the remote host IP,
a simpler way is to run the command:
# ssh-keygen -R REMOTE_IP

Linux Sign Generator

I came across this nice image generator of Linux penguine holding a sign with custom text, give it a try.

I recommend using 20pt pixels font with black color, like the one I used in this sed post.

You can also edit the link directly and just replace the text, for example, to generate a sign with above setting and the test "Linux Sign Generator" then the link would be:

Note: the character + used for spaces

Sed by Example

As you probably know, sed is a very powerful tool in Linux for filtering and transforming text.
I was looking for some commands with regular expression to process some configuration files in Linux, and I found this link that teaches sed by example, it's informative and easy to understand.

Note: If you read Arabic, sorry for the picture, no offence (I still find it funny) If you do not read Arabic, never mind.

Monday, September 20, 2010

Check multiple machines availability

Usually, administrators check machines liveness by sending ICMP echo request (aka ping request).

I found this tool very handy to check the liveness of multiple machines very quickly from command line. The tool is called fping, and it can be installed on Linux using this tar ball.

Define your machines IPs configuration file:
# cat machines.ips

Now, run fping to check the machines availability:
# fping -f machines.ips -r 1 -t 100 is alive is alive is alive is alive is alive is alive is alive is alive is alive is alive is unreachable

Script Header Generator

If you write a lot of scripts, you are aware of the importance of the scripts headers.
General, script header define the script language, usage, description, author..

I wrote this script, that generates a script header based on the template below, the script tries to identify the script language and usage automatically, in addition to some other fields, variables that need to be set manually are marked with "TBD" such as script description.

Feel free to change the AUTHOR and AUTHOR_EMAIL in the script to your own strings.
I found this scripts very useful to fix the header of hundreds of scripts I have that didn't include the appropriate header. Share and Enjoy.

Example, to generate the header of the script, run:
$ ./

#!/usr/bin/env python

# USAGE: ./ [--help|--version]
# DESCRIPTION: Tests how fast you can type A-Z
# REQUIREMENTS: python, finger
# NOTES: Dummy File
# AUTHOR: Ali Ayoub
# EMAIL: ali@ali.ali
# CREATED: 09.19.2010-19:05:39

Python Rocks.

The ARP Flux Problem

If you spent a lot of time trying to understand why a network interface in Linux fakes ARP replies on behalf of another interface, you may be facing the ARP flux effect! Yes, this is how it's called, and yes, you spend many hours debugging problems due to this weird behavior.

Surprisingly, this is a known behavior of Liux ARP module, when a a machine has multiple network interfaces, as stated in O'Reilly Book: Understanding Linux Network Internals:

The solution of the flux problem is through altering the system sysctl parameters, few websites (see references below) suggest how to make the suitable changes.

To make this process easier, you can use this script to set the right values into the procfs files.
To make the changes permanent (kept among reboots): the following can be added to your /etc/sysctl.conf file:
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

To flush the ARP table in Linux, use this helper script.


Friday, July 30, 2010

objdump parser

If you've been programming in Linux kernel, you must had the chance to debug kernel oops using objdump.
Normally, you invoke objdump with the name of the problematic driver, and then parse the output based on the information printed in the call trace of the kernel oops. Here is a nice example.

Among many hexadecimal numbers and function names that are printed in the call trace of the kernel oops, the most important piece of information is the function name and the offset: for example:

I wrote this script that receives the function name and offset as a pointer, and prints the source file name and line number that caused the oops.

For example:
# vnic_login_create_2+0xd1 drivers/net/mlx4_vnic/mlx4_vnic.ko

vnic_login_create_2+0xd1 = 0x6e3 => drivers/net/mlx4_vnic/vnic_data_main.c:210

Wednesday, July 21, 2010

"always on top" option for Windows

Always On Top Maker is a tiny interface less tool to make any window "always on top", or, make a topmost windows not always on top. How to use: Run AlwaysOnTopMaker.exe, then you can use 2 hot-keys, one is CTRL-ALT-T, is to switch the foreground application "always on top"/"not always on top", the other hotkey is CTRL-ALT-Q, is to exit Always On Top Maker.


Saturday, July 10, 2010

netconsole for redhat

Net-Console module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical. It can be used either built-in or as a module. As a built-in, netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. While this doesn't allow capture of early kernel panics, it does capture most of the boot process. it's very useful for kernel developers to dump kernel oops callstack (source).

The configuration in Red Hat is very simple:

On the monitor machine:
1. Edit /etc/sysconfig/syslog and include the line SYSLOGD_OPTIONS="-m 0 -r -x"
2. Run dmesg -n 8
3. Restart syslog service: /etc/init.d/syslog restart

On the development machine (that may have kernel oops):

1. Edit /etc/sysconfig/netconsole and include the line SYSLOGADDR=(monitor-machine-IP)
2. Restart netconsole service: /etc/init.d/netconsole restart

Now you can work on the development machine and watch its kernel messages on the monitor machine, I watch the development machine by running this command on the monitoring machine:

#tail -f /var/log/messages | grep (devel-machine-ip)

Below some useful bash scripts for your convenience:

If you're using remote console switch, you can check the console output by invoking the command:
# echo `date`: Hello from `whoami` > /dev/ttyS0

To test netconsole, you just need to trigger any kernel module that prints messages into /var/log/messages, for example:
# modprobe -qr tun; modprobe tun

If your running in Dom0 of Xen/KVM, the netconsole script get confused by the interface bridging, to fix this, you need to update /etc/init.d/netconsole:
First, find the corresponding peth interface of the netdev interface you want to use  for netconsole, normally if you're using eth0, then the corresponding inetrafce in Xen is peth0, then find the MAC address of the remote logger IP, for example if the remote sys logger is, find its MAC by running:
# ping -c 1 ; arp -n | grep
And finally, include the following two lines in /etc/init.d/netconsole:
 DEV=peth0 #your peth name here
 SYSLOGMACADDR=00:30:48:7D:DE:E4 #your MAC here
At this point you can use the scripts above to enable netconsole.

Thursday, July 08, 2010

SMS notification for Linux commands

Sometimes you need to run a Linux command that may take time (such as kernel compilation, virus scan, long test run..) instead of waiting in front of the monitor, you can do other stuff (e.g., good coffee) and you'll be notified when the command had finished with the results.

I have ATT cell phone, they support SMS2EMAIL service; simply send a message (plain text, no longer than 160 characters) to the address:
For example:

The Email address varies based on your carrier, check this website for worldwide cellular network list.

Now, we just need to run the command that takes time, and send a short text message with the report to notify us.. from a Linux machine with Email access, run this command:
(msg-body) | mail -s (msg-subject) (sms2email-address)

The message body can be the command output, or the command returned code, or anything else you want.

a) To report the returned code of an application, run:
gcc windows.c; echo "rc=$?" | mail -s "gcc report"
This will send you the returned code (rc) of the compilation command when its done, usually zero (rc=0) indicates a successful run.

b) To compile a Linux kernel and send an SMS when done with the output tail (last 120 bytes for example), run:
make 2>&1 | tail -c120 | mail -s "Kernel Compilation Status"

- You may need to adjust 'tail -c' flag to fit into 160 characters, see 'man tail' for more info.
- To redirect command error messages, use the suitable bash redirection methods.
- ATT sucks.

Wednesday, July 07, 2010

Fix Slow SSH Login Time

If your login times are really high, it may be that reverse DNS is not working correctly.

(a) Disable DNS in SSH daemon: go to /etc/ssh/sshd_config and include the line UseDNS no, then restart ssh service: /etc/init.d/sshd restart
(b) Or, fix your DNS configuration: in redhat, go to setup -> Network Configuration -> Edit DNS configuration -> and set the primary DNS and the search domain

Tuesday, January 12, 2010

Tips for better cscoping

Cscope is a developer's tool for browsing source code. It has an impeccable Unix pedigree, having been originally developed at Bell Labs back in the days of the PDP-11. Cscope was part of the official AT&T Unix distribution for many years, and has been used to manage projects involving 20 million lines of code! I use it a lot of work to manage browse the code of several kernels (any other editor I tried failed to do the job in reasonable time, cscope did handle it!)

Here are few tips for better cscoping:
- Display more pathname components in search results with -pN. By default, cscope only displays the basename of a given matching file. In large codebases, files in different parts of the source tree can often have the same name (consider main.c), which makes for confusing search results. By passing the -pN option to cscope at startup (or including -pN in the CSCOPEOPTIONS environment variable) -- where N is the number of pathname components to display -- this confusion can be eliminated. I've generally found -p4 to be a good middle-ground. Note that -p0 will cause pathnames to be omitted entirely from search results, which can also be useful for certain specialized queries.

- Use the built-in history mechanisms. You can quickly restore previous search queries by using ^b (control-b); ^f will move forward through the history. This feature is especially useful when performing depth-first exploration of a given function hierarchy. You can also use ^a to replay the most recent search pattern (e.g., in a different search field), and the > and < commands to save and restore the results of a given search. Thus, you could save search results prior to refining it using ^ (as per the previous tip) and restore them later, or restore results from a past cscope session.


Quick Start for lazy engineers:
find -name *.[ch] > cscope.files
cscope -bqk
cscope -d -p2
and 1.. 2..  3.. browse..

Wednesday, January 06, 2010

Kernel Debuging Best Tool!

Finally! kernel debugging best tool has been revealed