PARP VIPs very unreliable
I have a pretty simple network setup, WAN xx.xx.xx.2 (gateway xx.xx.xx.1, a router provided by ISP). LAN -> 192.168.168.0/24, DMZ 10.0.10.0/24. No VLANs, all physical NICs.
We have a class c block of static IPs xx.xx.xx.0/24 of which 2 are used up as the inside interface of the ISP router, and 1 used by the WAN interface of pfsense, so I've set up a number of VIPs on the WAN, and set up port forwarding in to the DMZ addresses xx.xx.xx.170:80 -> 10.0.10.170:80 for example… I have about 20 VIPs configured.
They will work for a few hours at a time, then a reboot of pfsense is required to get them to work again.. Sometimes only some of the VIPs stop working, sometimes its all of them. It seems like possibly an issue with ARP itself, like pfsense isn't broadcasting itself as the VIPs, or something. I don't have any other devices on this network segment (ISP router -> pfsense WAN port), so I don't have a good way to monitor the traffic, if necessary I could put something there to do that, but any ideas as to what might be causing this? seems like it is a pretty standard set up, anything I can try to force pfsense to respond to these addresses?
I am running pfsense 1.2.3.
After a bit more research, I can see the pfsense box replying to arp requests, however once the problem starts, it appears the ISP router is not accepting the arp replies? IE, you request something at xx.xx.xx.180 and the ISP router arps repeatedly, and pfsense replies repeatedly… but no traffic actually routes, like the ISP router isn't recognizing the reply? Is this a common issue with pARP? that routers have a hard time putting the same MAC in their arp table for multiple IPs?
I have 12 public IPs as virtual IPs using pARP and they have been working since 4 months without any issue at all. pARAP is pretty neat if you ask me.