Today I came across in interesting problem involving ARP broadcasts and the kernel's arp_ignore setting.
I'm in the process of deploying a little cluster for a busy website. We're using LVS directors from the Linux Virtual Server project to handle load-balancing and fail-over, and so I needed to configure the web-servers in my cluster not to send ARP broadcasts.
An ARP broadcasts is a special type of packet which is recognised by your network's switches. It's used to associate IP address with the MAC addresses. LVS directors need to share a single 'virtual' IP address with the other nodes on the network, so you need to configure those nodes NOT to send ARP broadcasts, else they will compete with the LVS director at the switch.
The problem was that I couldn't get the kernel's arp_ignore setting to stick. This is how you disable ARP broadcasts. You just need to place the following lines in /etc/sysctl.conf:
/etc/sysctl.conf
net.ipv4.conf.eth0.arp_ignore=3 net.ipv4.conf.eth0.arp_announce=2
And then reboot, or run the following command to make the change take effect immediately:
sysctl -p /etc/sysctl.conf
You can then check the settings just by outputting the appropriate files in /proc:
cat /proc/sys/net/ipv4/conf/eth0/arp_announce cat /proc/sys/net/ipv4/conf/eth0/arp_ignore
This was working just fine for me until I rebooted one of the nodes. I had expected that with the changes having been made in the sysctl.conf configuration file they would persist, but when I checked by repeating the above commands tge arp_ignore was back to it's default setting.
The culprit was Shorewall. It supports setting the arp_ignore setting through it's own configuration file. I hadn't set anything relating to arp_ignore, but because the setting was absent Shorewall was going ahead and over-writing the value I had set in sysctl.conf.
The solution was simply to configure the arp_ignore setting through Shorewall (as well as shown above) by adding the following to the interfaces file:
/etc/shorewall/interfaces
net eth0 detect dhcp,arp_ignore=3
Now the arp_ignore setting is correct after I have rebooted, and the switch knows where to route our packets.