MAC address craziness

michelangelo

…or ARP adventures.

I have dual wan setup, made following step-by-step the howto:

LAN -> 192.168.1.1
WAN1 -> 192.168.2.2, router1 192.168.2.1
WAN2 -> 192.168.3.2, router2 192.168.3.1

router1 MAC: xx:xx:xx:xx:xx:12
router2 MAC: xx:xx:xx:xx:xx:13

the two routers are connected directly to pfsense.

I had set up a Load balancing pool, two failovers and all the firewalling rules as in the tutorial, using 1.2RC4 on a Dell 860. The only difference is that I made up my own NAT rules, but identical to the default ones (i used pfctl -sn to see the autogenerated ones). I have sticky connections on. It worked flawlessly for a couple days, and I could test over and over the load balancer / failover.
Then, this morning, I decided to change the hostname from the default pfSense to something else (firewall). To my astonishment, I could not reach anymore the internet. No pings, nothing. But I could reach the routers (and ping them). This situation seemed to resolve by itself in 10/15 minutes. After another 15/20 minutes of working - whomp - nothing. Can reach the routers but no internet! AND I can reach the router's monitor, since all the links are marked UP in the load balancer page (I can ping the monitor, too, from my machine).
So I started dumping traffic between pfsense and router1. If I send a ping out I get this on the cable:

IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 1, length 64
IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 2, length 64
IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 3, length 64

but no response. Dumping traffic ON the router shows that it gets NO incoming traffic! How is this possible?? I had the answer "unpacking" a bit more the packets on the cable from pfsense to router1:

[WAN1_MAC] > xx:xx:xx:xx:xx:13, ethertype IPv4 (0x0800), length 98: 192.168.2.2 > [remote_host]: ICMP echo request, id 16778, seq 1, length 64
[WAN1_MAC] > xx:xx:xx:xx:xx:13, ethertype IPv4 (0x0800), length 98: 192.168.2.2 > [remote_host]: ICMP echo request, id 16778, seq 2, length 64
etc…

Yes it is as it seems: pfsense is sending the packet out of the right interface with the right source ip and mac but with the WRONG destination MAC! I bet I don't see anything on the router!
Obviously it is done the same for router2. The ARP tables remain always correct (the ones shown on the webif).
Any ideas?

Thanks,
Rodolfo

sai

really weird stuff (like this) usually means hardware problems: memory, power supply….

michelangelo

Mmh.. I'm inclined now to think it may be a problem with sticky conns… If I disable them everything works fine.
Is it some doc that talks about manual nat + sickies? Maybe I did something wrong

Cheers,
Rodolfo

sai

similar problem discussed here: http://forum.pfsense.org/index.php/topic,6204.0.html