Surprisingly, this is a known behavior of Liux ARP module, when a a machine has multiple network interfaces, as stated in O'Reilly Book: Understanding Linux Network Internals:
The solution of the flux problem is through altering the system sysctl parameters, few websites (see references below) suggest how to make the suitable changes.
To make this process easier, you can use this script to set the right values into the procfs files.
To make the changes permanent (kept among reboots): the following can be added to your /etc/sysctl.conf file:
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
To flush the ARP table in Linux, use this helper script.