Adding virtual IP completely breaks LAN/WAN connectivity(?!)
-
Hello!
I've got a hosted Proxmox bare metal environment where the provider gives me WANIP1 that connects directly to the Proxmox hardware. Through their services you purchase an additional IP and can follow instructions to essentially add that second IP (I'll just call it WANIP2) so it can be mapped to a pfSense VM running on Proxmox, and then have LAN clients run behind it. This is all setup and working perfectly. LAN clients can get Internet connectivity, and via a WAN-side firewall rule, I can remotely log into the pfSense and make changes. All good to go.
Now the customer has purchased a second block of additional IPs that are on a completely separate /28 network. If we try to add one of those virtual IPs and then setup a 1:1 NAT rule and forward some ports through to an internal host, internal/external connectivity breaks completely. LAN side can't see Internet, and I can no longer manage the firewall remotely.
A few moments ago I tried adding just any old virtual IP that is obviously not going to work (1.1.1.1) and everything completely breaks again. So I think it's less about those additional IPs at this point, and more about something I'm doing that is...breaking routing(?) once the additional IP is added.
Please help!
-
@RouterPounder I have added vips both on wan side and lan side and have never seen any issues. Just adding the vip shouldn't have anything to do with anything really.. So your saying it goes to hell with just adding the vip, or does it go to hell when you try and do some 1:1 nat?
-
@johnpoz Thanks for the reply. Yeah what makes this so, SO frustrating is this customer's old environment is setup nearly identical (the Proxmox+pfSense combo) and, like you, I've added a bunch of virtual IPs to environments like these with no issues. So the fact that adding a virtual IP is breaking things has me baffled. I even opened a ticket with the datacenter folks and they said everything is configured fine on their end. But like I said in original post, it doesn't matter what IP I add, it seems to break the world at some point.
I tried to recreate the steps with some timestamps and move slow to see if I could figure out exactly when things break, and I'm still puzzled. Here's that timeline:
- 11:29 a.m. - add just the virtual IP. Everything remains stable on LAN/WAN side.
- 11: 35 a.m. - add the 1:1 NAT from the new virtual IP to an internal host. Everything remains stable.
- 11:44 a.m. - started adding a port forward rule for HTTPS but DID NOT SAVE IT and suddenly lost connectivity. Checked and everything on LAN/WAN side is broken. LAN side (doesn't matter what VLAN) can't get out to internet, and I lose connection to manage the firewall externally.
So it's like at some point after the 1:1 NAT rule....SOMETHING is happening the makes everything go south.
Timeline for "fix":
- 11:50 a.m. - remove the 1:1 NAT rule. Things remain broken. Waited a while....then...
- 11:57 a.m. - remove the additional IP from the firewall. All connectivity fixes itself immediately.
Ugh!
-
Well I must still be on crazy pills. I completely nuked and rebuilt the pfSense from scratch. Started with the WAN side and "WANIP1" I was allocated for connectivity, and just left native VLAN that came along with install to keep things super simple. On the LAN side network connectivity is working great. I add a second virtual IP (valid or not), and the minute I put in the 1:1 NAT rule and/or port forward, connectivity is completely broken. I'm wondering if I've got something outside of the VM going on but I can't think of what that would be!
-
I figured it out! This Proxmox host is running on OVHCloud. When setting up the networking, you need to order an additional IP and assign a virtual MAC to it for the WAN side. Any extra IPs must also use that virtual MAC. Once I did that, everything worked perfectly. I'm still not entirely sure why the whole network would crash without the virtual MAC in place, but hey, no complaints here—it's working now!
