MAC address craziness



  • …or ARP adventures.

    I have dual wan setup, made following step-by-step the howto:

    LAN -> 192.168.1.1
    WAN1 -> 192.168.2.2, router1 192.168.2.1
    WAN2 -> 192.168.3.2, router2 192.168.3.1

    router1 MAC: xx:xx:xx:xx:xx:12
    router2 MAC: xx:xx:xx:xx:xx:13

    the two routers are connected directly to pfsense.

    I had set up a Load balancing pool, two failovers and all the firewalling rules as in the tutorial, using 1.2RC4 on a Dell 860. The only difference is that I made up my own NAT rules, but identical to the default ones (i used pfctl -sn to see the autogenerated ones). I have sticky connections on. It worked flawlessly for a couple days, and I could test over and over the load balancer / failover.
    Then, this morning, I decided to change the hostname from the default pfSense to something else (firewall). To my astonishment, I could not reach anymore the internet. No pings, nothing. But I could reach the routers (and ping them). This situation seemed to resolve by itself in 10/15 minutes. After another 15/20 minutes of working - whomp - nothing. Can reach the routers but no internet! AND I can reach the router's monitor, since all the links are marked UP in the load balancer page (I can ping the monitor, too, from my machine).
    So I started dumping traffic between pfsense and router1. If I send a ping out I get this on the cable:

    IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 1, length 64
    IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 2, length 64
    IP 192.168.2.2 > [remote_host]: ICMP echo request, id 41528, seq 3, length 64

    but no response. Dumping traffic ON the router shows that it gets NO incoming traffic! How is this possible?? I had the answer "unpacking" a bit more the packets on the cable from pfsense to router1:

    [WAN1_MAC] > xx:xx:xx:xx:xx:13, ethertype IPv4 (0x0800), length 98: 192.168.2.2 > [remote_host]: ICMP echo request, id 16778, seq 1, length 64
    [WAN1_MAC] > xx:xx:xx:xx:xx:13, ethertype IPv4 (0x0800), length 98: 192.168.2.2 > [remote_host]: ICMP echo request, id 16778, seq 2, length 64
    etc…

    Yes it is as it seems: pfsense is sending the packet out of the right interface with the right source ip and mac but with the WRONG destination MAC! I bet I don't see anything on the router!
    Obviously it is done the same for router2. The ARP tables remain always correct (the ones shown on the webif).
    Any ideas?

    Thanks,
    Rodolfo



  • really weird stuff (like this) usually means hardware problems: memory, power supply….



  • Mmh.. I'm inclined now to think it may be a problem with sticky conns… If I disable them everything works fine.
    Is it some doc that talks about manual nat + sickies? Maybe I did something wrong

    Cheers,
    Rodolfo




Locked