Connectivity lost, comes back only after NAT change



  • SETUP
    WAN : x.y.z.1/29 net, pfsense uses x.y.z.1, WAN gw is x.y.z.6 and we also use some NAT'ed IPs (x.y.z.2, x.y.z.3, see below).
    LAN  : a.b.c.0/24
    in pfSense 2.1 virtual IP is configured to x.y.z.3 and 1:1 NAT'ed to a.b.c.3 for our internal server.

    PROBLEM
    Now one day we lost internet due to ISP issues. When it comes back, NAT'ed server a.b.c.3 has no internet access and also is not accessible from internet.
    Tried to resolve this, but run out of ideas, really. OK, replaced firewall, which was m0n0wall at the time with pfSense.
    Configured everything, and worked for a week.
    Tonight we had to switch off pfSense for an hour.
    When it comes back, the same issue - server has no internet.  >:(
    All other hosts are working fine, has no problems to get through firewall.
    After enless server and switch reboots I finally changed virtual IP and NAT 1:1 on pfSense bot from x.y.z.3 to x.y.z.4
    Voilá! Internet on server is back.  ;D
    Now I change again x.y.z.4 back to x.y.z.3 as it was initially. No problem, internet is stil there.  ::)

    What the hell is going on here? Makes me really wonder why it did happen on both m0n0wall and pfSense…

    Thanks for you time,
    shpokas



  • You're causing issues with your upstream ARP cache by switching devices around. Using IP alias type VIPs commonly helps that situation since they send a gratuitous ARP which can update the upstream ARP cache. Proxy ARP strictly responds to ARP requests which means you'll have to wait until the upstream ARP cache times out when switching hardware, which can be several hours. Or if your upstream router is your modem or otherwise accessible to you, power cycle it after changing devices.



  • Not sure. Problem was definetely there already before we switched devices. Also, swhitching devices somehow helped.
    Seems that broken connectivity (either upstream device or pfSense down) caused 1:1 NAT to stop working. But why?