Virtual IP strangeness
I'm trying to migrate some existing web servers to a pfSense setup. Some of the customers have their own ip address, and I just attempted to move one of those from a live server to being a virtual ip address on pfSense that NATs to an internal computer.
Due to some problems with the web server configurations on the new internal computer, we switched the sites back by removing the virtual ip address from pfSense and restoring that public ip address on the public web server.
After that we waited several minutes but the sites didn't come back up and the ip address wasn't pingable. I rebooted pfSense "just to be sure". Figuring a provider ARP cache, we gave it a few more minutes. Eventually I noticed that pfSense was filtering the ICMP attempts I was sending to the ip address. I added a rule to allow TCP to the ip address, and suddenly the websites became available again.
I don't understand why / how pfSense was able to intercept the data destined for the public ip address on the public server. Is this expected behavior, can someone please explain?
Upstream ARP cache. The IP won't move back until it's cleared or times out, which takes several hours by default on every router. 4 hours on Cisco, similar on others.