CARP Troubleshooting
-
Heyo,
I'd like to ask for some assistance in troubleshooting a carp issue that I've come across.
I have a configuration like this:
(note the us-16-150w is the same switch, I'm just separating the modem connection via vlan 999)
Both pfsense boxes are a VM on separate proxmox nodes
The nodes are connected to the us-16-150 via a trunk port that allows all vlans
The pfsense VMs (2n1 and 2n2) are connected to the hosts with 2 vtnet ports, to the same software bridge
vtnet0 (WAN) is tagged by proxmox as 999
vtnet1 (LAN) is not tagged at allI have configured HA/CARP/VIPs as suggested via the documentation
I'm using a /29 WAN ip block
Previously this has worked when connected to a BGW320 in pass-thru
I'm now connected to a DOCSIS 3.1 modem, fiber isn't available at new address :(WAN interfaces are assigned in sequential order
2n1 is .70
2n2 is .71
CARP vip is .72I have 2 issues:
on initial startup 2n1 will assume master, but if I failover to 2n2, and go back I start to lose packets
If I contact the ISP they mention seeing .72 "bounce" between the WAN interface MAC address and the CARP interface mac address
This behavior causes the CMTS (maybe the modem, it's been a bit since I spoke to them) to drop almost all packetsIf I enable hybrid outbound NAT, new outbound connections fail
I've tried enabling it, rebooting the pfsense nodes and modem, but didn't see a difference until I disabled the featureI'm assuming one or the other here, I should be able to gather information on
I do have a machine that can connect to vlan 999
I tried to do a capture with wireshark, but I didn't see the behavior that the ISP mentioned in ARP
(this has been mentioned on 2 different calls with them, with different technicians)I'm unfortunately, unsure about the hybrid nat issue, I'm not seeing anything in the logs, but NAT is a bit outside my wheelhouse
More than welcome to any ideas on that front. -
@Deputize2180
It seems you are facing two different issues, that can be related but you should really begin with the packet loss and failover issue.
Until this is solved, messing with nat creates noise.Can you explain what tagging has to do with the wan?
I assume the modem isn't expecting any tagged traffic to begin with.
Just a /29 with different mac addresses bound to ip's/
As long as the ports as marked as access, vlans are out of the way.Have you tried a unicast carp connection?
can you elaborate on
@Deputize2180 said in CARP Troubleshooting:
they mention seeing .72 "bounce" between the WAN interface MAC address and the CARP interface mac address
by carp you mean virtual ip? Which wan interface pf1 or pf2?
-
@netblues
Hey thanks for the reply,
And yeah I definitely missed some info hereThe modem is connected to a switch port with native vlan set to 999
And blocking all tagged vlans
(not a fan of this not just being named access port in ui)The pfsense VMs are connected to the network with vtnet0 and vtnet1 on the same bridge
Because the bridge connected nic, is connected to the switch via a trunk port that is allowing all vlansVtnet0 (on both VMs) and the modem are the only things on vlan 999
I’m definitely wanting to sort the failover issue first, I’m assuming maybe some of the behavior here may be leading to the NAT issue
I have not tried Unicast CARP yet, I’ll have to do some reading I was unaware that was an option, and I think it might help.
And lastly you were correct I meant VIP
Once I failover and back, the ISP reports seeing the VIP (.72) bounce between the vtnet0 MAC address and the vip interface MAC address. This happens with the master node once it enters this state. (If I put 2n1 into CARP maintenance mode, it happens with 2n2-vtnet0 and the vip interface) -
Unicast is most probably the only viable test, but I doubt it will fix things.
Most probably the isp modem has issues with carp and will never work properly.
I'm not aware of any other tunable options too. (and I do hope I'm wrong)