SG-3100 disconnects every 20min w/ Cisco AnyConnect VPN client
-
@stephenw10
Correct, no difference at all. I've rebooted the router in-between changes of these settings as well.I've checked the state table, but I'm unsure of exactly what I'm looking for. I noticed a couple connection issues that likely pertain to this issue, but I'm unsure of how to interpret and proceed. Obfuscated snippet below:
WAN IP: 100.100.100.100 LAN IP: 192.168.1.10 Corp IP 1: 150.10.10.10 Corp IP 2: 150.20.20.20 Corp IP 3: 150.30.30.30 WAN tcp 100.100.100.100:19020 (192.168.1.10:49197) -> 150.10.10.10:443 SYN_SENT:CLOSED 3 / 0 192 B / 0 B WAN tcp 100.100.100.100:42692 (192.168.1.10:49198) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B 99_VLAN tcp 192.168.1.10:49372 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_VLAN tcp 192.168.1.10:49376 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B
-
I expect to see the states open with traffic both ways if the anyconnect tunnel is up and carrying traffic.
If usually uses UDP 443 or falls back to TCP 443 if that's not available. It could be possible to use a custom port though I'm not aware of that.Steve
-
The actual state table shows other states with traffic going both ways for the VPN connection.
The above was only a snippet of some of the states; ones that were CLOSED and didn't look to be fully connected (0 bytes sent). I believe these states are what are causing the disconnects after 20min (ie. if these aren't connecting within 20min, kill the active VPN connection and reconnect).
My question is: using these 4 CLOSED states as the potential cause of the problem, what would you suggest I do to further debug? I'm taking a stab in the dark but could it be potential port forwarding issues? Would I set up a port forward for 9997 to my laptop?
Thanks!
-
Each outbound connection from your laptop will create a state on the internal interface and a state on the WAN including NAT.
Those 4 closed states are all different though. Different source ports on each one. Was there a matching state for each that was still open?
What I expect to see is a state opened when the VPN connects and held open at least until the tunnel rekeys. If for some reason it's not opening states at that point in one of the interfaces that would obviously be a problem.Steve
-
@stephenw10 it doesn't look like there were any matching open states for that example. Here's the entire state table of the above example (IPs obfuscated).
150.90.90.90 is the IP that the AnyConnect client is set to connect (https://VPN-domain.com in OP).
WAN IP: 100.100.100.100 LAN IP: 192.168.1.10 VPN IP: 150.90.90.90 Corp IP 1: 150.10.10.10 Corp IP 2: 150.20.20.20 Corp IP 3: 150.30.30.30 interface protocol connection state conns bytes 99_GUEST tcp 192.168.1.10:49327 -> 150.40.40.40:9997 CLOSED:SYN_SENT 1 / 0 64 B / 0 B 99_GUEST tcp 192.168.1.10:49329 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49332 -> 150.20.20.20:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49345 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49352 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49358 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49367 -> 150.20.20.20:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49372 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49376 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49378 -> 150.20.20.20:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49380 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49385 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49387 -> 150.20.20.20:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49389 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49391 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49399 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49405 -> 150.30.30.30:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49408 -> 150.20.20.20:9997 CLOSED:SYN_SENT 8 / 0 512 B / 0 B 99_GUEST tcp 192.168.1.10:49394 -> 150.40.40.40:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49397 -> 150.20.20.20:9997 CLOSED:SYN_SENT 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49337 -> 70.70.70.70:443 CLOSING:ESTABLISHED 78 / 79 9 KiB / 19 KiB 99_GUEST tcp 192.168.1.10:49338 -> 70.70.70.70:443 CLOSING:ESTABLISHED 78 / 79 9 KiB / 19 KiB 99_GUEST tcp 192.168.1.10:49339 -> 70.70.70.70:443 CLOSING:ESTABLISHED 78 / 80 9 KiB / 19 KiB 99_GUEST tcp 192.168.1.10:49364 -> 150.60.60.60:443 CLOSING:ESTABLISHED 121 / 120 145 KiB / 13 KiB 99_GUEST tcp 192.168.1.10:49368 -> 150.60.60.60:443 CLOSING:ESTABLISHED 22 / 31 7 KiB / 8 KiB 99_GUEST tcp 192.168.1.10:49400 -> 150.50.50.50:443 CLOSING:ESTABLISHED 14 / 23 2 KiB / 6 KiB 99_GUEST tcp 192.168.1.10:49336 -> 70.70.70.70:443 CLOSING:ESTABLISHED 78 / 80 9 KiB / 19 KiB 99_GUEST tcp 192.168.1.10:49340 -> 70.70.70.70:443 CLOSING:ESTABLISHED 106 / 97 11 KiB / 25 KiB 99_GUEST tcp 192.168.1.10:49401 -> 150.50.50.50:443 CLOSING:ESTABLISHED 435 / 600 24 KiB / 846 KiB WAN tcp 100.100.100.100:64063 (192.168.1.10:49156) -> 2.2.2.23:5223 ESTABLISHED:CLOSING 39 / 35 5 KiB / 6 KiB WAN tcp 100.100.100.100:50569 (192.168.1.10:49228) -> 82.82.82.82:443 ESTABLISHED:CLOSING 17 / 21 2 KiB / 8 KiB WAN tcp 100.100.100.100:24749 (192.168.1.10:49246) -> 84.84.84.84:443 ESTABLISHED:CLOSING 28 / 33 6 KiB / 8 KiB WAN tcp 100.100.100.100:13815 (192.168.1.10:49256) -> 84.84.84.84:443 ESTABLISHED:CLOSING 321 / 397 43 KiB / 406 KiB WAN tcp 100.100.100.100:43609 (192.168.1.10:49261) -> 70.70.70.70:443 ESTABLISHED:CLOSING 47 / 54 5 KiB / 30 KiB WAN tcp 100.100.100.100:29558 (192.168.1.10:49264) -> 70.70.70.70:443 ESTABLISHED:CLOSING 50 / 56 6 KiB / 23 KiB WAN tcp 100.100.100.100:41905 (192.168.1.10:49265) -> 70.70.70.70:443 ESTABLISHED:CLOSING 48 / 54 6 KiB / 30 KiB WAN tcp 100.100.100.100:23005 (192.168.1.10:49268) -> 151.101.0.106:443 ESTABLISHED:CLOSING 19 / 26 2 KiB / 6 KiB WAN tcp 100.100.100.100:40183 (192.168.1.10:49271) -> 70.70.70.70:443 ESTABLISHED:CLOSING 87 / 84 11 KiB / 37 KiB WAN tcp 100.100.100.100:8542 (192.168.1.10:49274) -> 72.72.72.72:443 ESTABLISHED:CLOSING 18 / 25 2 KiB / 7 KiB WAN tcp 100.100.100.100:50966 (192.168.1.10:49283) -> 150.60.60.60:443 ESTABLISHED:CLOSING 20 / 27 6 KiB / 7 KiB WAN tcp 100.100.100.100:9312 (192.168.1.10:49294) -> 150.60.60.60:443 ESTABLISHED:CLOSING 75 / 74 87 KiB / 10 KiB WAN tcp 100.100.100.100:40462 (192.168.1.10:49308) -> 35.35.35.35:443 ESTABLISHED:CLOSING 239 / 100 323 KiB / 11 KiB WAN tcp 100.100.100.100:13052 (192.168.1.10:49316) -> 11.11.11.11:443 ESTABLISHED:CLOSING 17 / 24 2 KiB / 8 KiB WAN tcp 100.100.100.100:14594 (192.168.1.10:49322) -> 150.50.50.50:443 ESTABLISHED:CLOSING 599 / 1.821 K 33 KiB / 2.57 MiB WAN tcp 100.100.100.100:21254 (192.168.1.10:49323) -> 24.24.24.24:443 ESTABLISHED:CLOSING 10 / 19 2 KiB / 7 KiB WAN tcp 100.100.100.100:28927 (192.168.1.10:49324) -> 150.70.70.70:443 ESTABLISHED:CLOSING 13 / 18 3 KiB / 6 KiB WAN tcp 100.100.100.100:61747 (192.168.1.10:49325) -> 150.80.80.80:443 ESTABLISHED:CLOSING 15 / 24 2 KiB / 11 KiB WAN tcp 100.100.100.100:31961 (192.168.1.10:49337) -> 70.70.70.70:443 ESTABLISHED:CLOSING 78 / 79 9 KiB / 19 KiB WAN tcp 100.100.100.100:8863 (192.168.1.10:49338) -> 70.70.70.70:443 ESTABLISHED:CLOSING 78 / 79 9 KiB / 19 KiB WAN tcp 100.100.100.100:18026 (192.168.1.10:49339) -> 70.70.70.70:443 ESTABLISHED:CLOSING 78 / 80 9 KiB / 19 KiB WAN tcp 100.100.100.100:3256 (192.168.1.10:49364) -> 150.60.60.60:443 ESTABLISHED:CLOSING 121 / 120 145 KiB / 13 KiB WAN tcp 100.100.100.100:44448 (192.168.1.10:49368) -> 150.60.60.60:443 ESTABLISHED:CLOSING 22 / 31 7 KiB / 8 KiB WAN tcp 100.100.100.100:62411 (192.168.1.10:49400) -> 150.50.50.50:443 ESTABLISHED:CLOSING 14 / 23 2 KiB / 6 KiB WAN tcp 100.100.100.100:18795 (192.168.1.10:49262) -> 70.70.70.70:443 ESTABLISHED:CLOSING 50 / 54 6 KiB / 21 KiB WAN tcp 100.100.100.100:19285 (192.168.1.10:49336) -> 70.70.70.70:443 ESTABLISHED:CLOSING 78 / 80 9 KiB / 19 KiB WAN tcp 100.100.100.100:41344 (192.168.1.10:49340) -> 70.70.70.70:443 ESTABLISHED:CLOSING 106 / 97 11 KiB / 25 KiB WAN tcp 100.100.100.100:38858 (192.168.1.10:49401) -> 150.50.50.50:443 ESTABLISHED:CLOSING 435 / 600 24 KiB / 846 KiB WAN tcp 100.100.100.100:9445 (192.168.1.10:49217) -> 70.70.70.70:443 ESTABLISHED:ESTABLISHED 67 / 72 7 KiB / 36 KiB WAN tcp 100.100.100.100:13222 (192.168.1.10:49224) -> 70.70.70.70:443 ESTABLISHED:ESTABLISHED 85 / 88 8 KiB / 32 KiB WAN tcp 100.100.100.100:49576 (192.168.1.10:49201) -> 195.195.195.195:443 ESTABLISHED:ESTABLISHED 11 / 12 2 KiB / 6 KiB 99_GUEST tcp 192.168.1.10:49411 -> 150.90.90.90:443 ESTABLISHED:ESTABLISHED 56 / 55 6 KiB / 15 KiB WAN tcp 100.100.100.100:10923 (192.168.1.10:49411) -> 150.90.90.90:443 ESTABLISHED:ESTABLISHED 56 / 55 6 KiB / 15 KiB WAN tcp 100.100.100.100:39287 (192.168.1.10:49320) -> 150.90.90.90:443 ESTABLISHED:ESTABLISHED 16 / 19 6 KiB / 6 KiB WAN tcp 100.100.100.100:22620 (192.168.1.10:49363) -> 150.80.80.80:443 ESTABLISHED:FIN_WAIT_2 15 / 15 2 KiB / 8 KiB WAN tcp 100.100.100.100:12169 (192.168.1.10:49381) -> 41.41.41.41:443 ESTABLISHED:FIN_WAIT_2 33 / 15 34 KiB / 5 KiB WAN tcp 100.100.100.100:62159 (192.168.1.10:49290) -> 150.50.50.50:443 ESTABLISHED:FIN_WAIT_2 32 / 32 4 KiB / 35 KiB 99_GUEST tcp 192.168.1.10:49363 -> 150.80.80.80:443 FIN_WAIT_2:ESTABLISHED 15 / 15 2 KiB / 8 KiB 99_GUEST tcp 192.168.1.10:49381 -> 41.41.41.41:443 FIN_WAIT_2:ESTABLISHED 33 / 15 34 KiB / 5 KiB 99_GUEST tcp 192.168.1.10:49382 -> 190.190.190.190:443 FIN_WAIT_2:FIN_WAIT_2 14 / 15 3 KiB / 1 KiB WAN tcp 100.100.100.100:32103 (192.168.1.10:49382) -> 190.190.190.190:443 FIN_WAIT_2:FIN_WAIT_2 14 / 15 3 KiB / 1 KiB 99_GUEST tcp 192.168.1.10:49407 -> 150.90.90.90:443 FIN_WAIT_2:FIN_WAIT_2 28 / 27 8 KiB / 13 KiB WAN tcp 100.100.100.100:15685 (192.168.1.10:49407) -> 150.90.90.90:443 FIN_WAIT_2:FIN_WAIT_2 28 / 27 8 KiB / 13 KiB 99_GUEST tcp 192.168.1.10:49402 -> 180.180.180.180:443 FIN_WAIT_2:FIN_WAIT_2 18 / 20 5 KiB / 8 KiB WAN tcp 100.100.100.100:12399 (192.168.1.10:49402) -> 180.180.180.180:443 FIN_WAIT_2:FIN_WAIT_2 18 / 20 5 KiB / 8 KiB 99_GUEST tcp 192.168.1.10:49403 -> 3.3.3.3:443 FIN_WAIT_2:FIN_WAIT_2 13 / 9 2 KiB / 6 KiB WAN tcp 100.100.100.100:50172 (192.168.1.10:49403) -> 3.3.3.3:443 FIN_WAIT_2:FIN_WAIT_2 13 / 9 2 KiB / 6 KiB 99_GUEST udp 192.168.1.10:55072 -> 192.168.1.1:53 MULTIPLE:MULTIPLE 2 / 2 140 B / 262 B 99_GUEST udp 192.168.1.10:61843 -> 150.90.90.90:443 MULTIPLE:MULTIPLE 1.397 K / 956 553 KiB / 372 KiB WAN udp 100.100.100.100:15097 (192.168.1.10:61843) -> 150.90.90.90:443 MULTIPLE:MULTIPLE 1.397 K / 956 553 KiB / 372 KiB WAN tcp 100.100.100.100:59294 (192.168.1.10:49182) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:42144 (192.168.1.10:49187) -> 150.10.10.10:443 SYN_SENT:CLOSED 2 / 0 128 B / 0 B WAN tcp 100.100.100.100:58814 (192.168.1.10:49188) -> 150.10.10.10:443 SYN_SENT:CLOSED 2 / 0 128 B / 0 B WAN tcp 100.100.100.100:40975 (192.168.1.10:49189) -> 150.10.10.10:443 SYN_SENT:CLOSED 2 / 0 128 B / 0 B WAN tcp 100.100.100.100:22169 (192.168.1.10:49190) -> 150.10.10.10:443 SYN_SENT:CLOSED 2 / 0 128 B / 0 B WAN tcp 100.100.100.100:23190 (192.168.1.10:49196) -> 150.10.10.10:443 SYN_SENT:CLOSED 3 / 0 192 B / 0 B WAN tcp 100.100.100.100:19020 (192.168.1.10:49197) -> 150.10.10.10:443 SYN_SENT:CLOSED 3 / 0 192 B / 0 B WAN tcp 100.100.100.100:42692 (192.168.1.10:49198) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:27054 (192.168.1.10:49200) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:42574 (192.168.1.10:49293) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:60912 (192.168.1.10:49304) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:42655 (192.168.1.10:49307) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:48261 (192.168.1.10:49309) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:11488 (192.168.1.10:49321) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:25840 (192.168.1.10:49327) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:10674 (192.168.1.10:49329) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:35184 (192.168.1.10:49332) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:61260 (192.168.1.10:49345) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:18090 (192.168.1.10:49352) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:63274 (192.168.1.10:49358) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:47537 (192.168.1.10:49367) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:50689 (192.168.1.10:49372) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:63540 (192.168.1.10:49376) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:53611 (192.168.1.10:49378) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:56886 (192.168.1.10:49380) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:27042 (192.168.1.10:49385) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:7531 (192.168.1.10:49387) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:61761 (192.168.1.10:49389) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:14040 (192.168.1.10:49391) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:35582 (192.168.1.10:49399) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:40819 (192.168.1.10:49405) -> 150.30.30.30:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:39882 (192.168.1.10:49408) -> 150.20.20.20:9997 SYN_SENT:CLOSED 8 / 0 512 B / 0 B WAN tcp 100.100.100.100:53085 (192.168.1.10:49255) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:60795 (192.168.1.10:49310) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:42614 (192.168.1.10:49394) -> 150.40.40.40:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B WAN tcp 100.100.100.100:45525 (192.168.1.10:49397) -> 150.20.20.20:9997 SYN_SENT:CLOSED 9 / 0 576 B / 0 B 99_GUEST tcp 192.168.1.10:49361 -> 2.2.2.2:5223 TIME_WAIT:TIME_WAIT 26 / 18 4 KiB / 5 KiB WAN tcp 100.100.100.100:21405 (192.168.1.10:49361) -> 2.2.2.2:5223 TIME_WAIT:TIME_WAIT 26 / 18 4 KiB / 5 KiB
Also, I tried experimenting with unchecking "disable firewall scrub" and "ip do-not-fragment compatibility" (suggestions from some additional threads I found) but neither worked.
-
I see two matched pairs of states from source ports 61843 and 49411and one unmatched state from 49320. The internal state has closed for that but there is almost no traffic on it.
Is the client reporting a failed connection at that point?
Steve
-
@stephenw10 yes, this state was captured moments after the failed connection occurred.
-
Hmm, well nothing there looks unusual except maybe that state on WAN only.
Might need a packet capture to see what's failing there. I don't see any other reports of that mode of failure.Steve
-
Following up on this in-case others have this same issue:
It turns out there was a filtering issue, but it wasn't from pfSense. I have a Unifi AC-Pro WAP which has a filtering option called Multicast and Broadcast Filtering which is enabled by default on guest networks. Disabling this feature resolved the issue.
-
Nice catch! Hard to imagine what the Anyconnect client needed that would be blocked by such a filter. If it was filtering as expected at least.
Steve