TCP Keepalives failing over NAT
-
I have analyzed the packet traces and this is what I see. Take note that the packet captures were done on all lifelines shown here EXCEPT for the PC, but that is not important as the PC never has issues not receiving packets / sending packets. The issue seems to be on the LAN segment between SERVER and FW2LAN1.
-
"The issue seems to be on the LAN segment between SERVER and FW2LAN1."
Would make sense.. Can you check the layer 1 between, maybe move ports on your switch.. What specific switch you running and firmware? We had a really odd bug in cisco switch one time where all would work great and then all of sudden switch would not pass on packets that were inside specific vlan. You would see it enter the switch, but not leave..
Oddest freaking thing - took a bit to track it down. and work around until we updated the firmware was just add an svi on that vlan - even if not being used, since it was just a layer 2 vlan.
If you can sniff right on your switch on this specific vlan might give you some insight you should see the packets twice, once when they enter the switch and then again when they leave the switch.
-
Will see what I can do once my client gets back to me with the switch's details.
-
So the switch is a Cisco SG200-18, and was on firmware 1.3.7.18 (I was told it was latest, clearly it was not). I just updated it to 1.4.8.6 - so I guess I have to wait and see. Just weird that this is the only issue we have on this network segment - nothing else is broken (AFAIK).
PS: There are no VLANs configured on this switch - all nodes use the default VLAN. And as far as I can tell, even though this is a managed switch, nothing non standard has been configured.
-
So no - the firmware update fixed nothing, in fact, it seems worse now - these issues appear once every 1 - 2 hours now. I cannot do a packet capture on the switch. Not sure how to troubleshoot this further.
-
Any ideas?
-
Here is the thing your sniffing on the devices connected to the switch.. And you see 1 device send the packets, and the other device does not get the packets.. That really really points to the switch.. You don't have any patch panels between the devices right.
you have
device1 –- cable --- switch --- cable --- device2If your sniffing on 1 and 2, and you see 1 send but 2 does not get.. Its the switch or the cables - and have never seen a cable drop packets just now and then.. If they are bad they are normally bad! all the time. But to rule it out you could change the cables.
Have you changed the ports the devices are connected to on the switch?
-
you have
device1 –- cable --- switch --- cable --- device2That is my understanding, yes (I am remote troubleshooting with someone's help)
Have you changed the ports the devices are connected to on the switch?
Not yet, but I have asked the guy to attach a different Cisco switch to that switch, and have the server and client attached to the "new" switch - this should rule out the switch. Waiting to hear back from him.
-
yup that would do it as well… Let us know what happens! Suggest use new cables as well when you connect to new switch.
-
So after 1 week of having just the firewall and the server on the new switch, we see no transport related errors. So it is probably a very weird error inside that switch. We will be replacing it and see if that fixes the issue.
-
Glad to hear you have found something.. So taking no cisco support on the switch so you could open a tac case with cisco? sg200 pretty cheap.. Prob just throw it out ;)
-
No such luck I am afraid. The "new" switch is a SG200-26. Would love to understand what went wrong in that switch though. Almost like a messed up ARP table?
