XG-7100 - Internal switch - packet loss
I have two XG-7100 setup in and HA configuration and am using the internal switches to connect with two different ISPs.
I have an unexplainable (so far) packet loss problem ONLY from FWB with one just one of the ISPs.
I changed out FWB for another XG-7100 - loaded up config and same issue.
Packets from FWB to either ISP must traverse a cross connect between FWA & FWB on port 8. Details are below on switch setup and connections. Same result when ISP2 is connected to either firewalls internal switch.
I am using the internal switches for the ISP connections, a separate vlan and lagg is setup for each ISP.
Internal SW ports 1-4 VLAN 4090 - ISP1 connected to port 1
Internal SW ports 5-7 VLAN 4089 - ISP2 connected to port 5
Internal SW port 8, VLAN 1, 4090, 4089 - xover to FWB port 8
Internal SW ports 1-4 VLAN 4090 -
Internal SW ports 5-7 VLAN 4089 - ISP2 - when connected port 5 here also has loss. FWA can ping ISP2 GW with zero loss.
Internal SW port 8, VLAN 1, 4090, 4089 - xover to FWA port 8
WAN interfaces are laggs connecting to each switch via ix2 & ix3 soc uplink to internal switch ports 9 & 10.
WAN - lagg0.4090 - ISP1
WAN2 - lagg0.4089 - ISP2
FWA - No packetloss when ping ISP1 upstream GW
FWA - No packetloss when ping ISP2 upstream GW - even when ISP2 is connected to port 5 of FWB
FWB - No packetloss when ping ISP1 upstream GW - key piece, this always works, its only ISP2 from FWB that fails.
FWB - Always packet loss when ping to ISP2 GW or beyond
FWA can always ping with no loss to ISP2 GW and beyond
Running tcpdump for two target hosts on FWB shows all outbound packets but 10-15% of ICMP reply never seen.
tcpdump -i lagg0.4089 host ISP2GW
tcpdump -i lagg0.4089 host 184.108.40.206 - static route to force over ISP2
I had ISP do some testing and when sourcing from their GW to my FWB had loss. When they sourced from another router on their network no loss.
I'm thinking its the firewall since ISP2 can also ping FWA with no issue.
Can you change the IP FWB is using the ISP2 and retest? That's the big difference there between the two devices, the source IP.
The fact the ISP could ping the FWB IP from a different router on their network with no loss even though that traffic still has to go through the gateway is suspicious. Have I understood that correctly?
Can you ping out from FWB via ISP2 to some target other than the gateway? Do you see loss then?
Yes I did try changing the ISP2 address on FWB, same result so set it back to the original IP.
Confirming that when ISP sourced ping to my IP from an address other than the directly connect router there was no loss.
When I ping the GW/router address or beyond (220.127.116.11) same behavior, 10+% loss.
I have another office with an identical setup, different ISPs though.
Its really weird, when I source a ping out the FWB ISP2 interface I don't get all the return traffic. With tcpdump from the command line of FWB I see all the outbound request sequences. Whether or not it made it out of the internal switch I can't tell. Anyway to mirror traffic to another port with the internal Marvell switch? I did a ping to one of my other locations and did a tcdump capture at the remote end and the echo requests never arrived for the "dropped" pings.
Another thing that works with no loss is a host behind FWB on than LAN can ping 18.104.22.168 and the ISP2 router with no loss. Regular ping (1 per sec) and flood ping. It seems to be isolate to pings sourced from FWB or sourced from ISP router IP to my FWB ISP2 addr. Everything else tested works with no loss.
Ok I would connect something completely different to the ISP2 connection and use the same FWB IP. See if there is still loss from there.
When I had just flipped the IPs between FWA and FWB yesterday I still had loss on FWB but did not test from FWA perhaps.
I just moved FWB IP to FWA and put another free address on FWB. Loss now on FWA.
Going to be calling the ISP again.
I did some other crazy stuff to route from FWB to FWA to get to 22.214.171.124, did packet captures and proved FWB was indeed sending request packets for the dropped pings being reported.
Thanks for the suggestion.