Cisco AnyConnect VPN behind a pfSense 2.4.5

[Alex the firewall]

Hello

First time posting here, I signed up to ask this question after a good search around..

I also use Cisco AnyConnect VPN on a work laptop behind a pfsense firewall at home and the scenario is very similar to a post on this forum i.e. if I use a simple device such as an ISP provided modem/router, 4G nighthawk or hotspot'ing to the phone all is good..

What's my issue? When I connect to to my work network the connection is successful but after a short time of ~60sec the connection disconnects and the reconnects to the same VPN peer and 9 from 10 times stays connected for hours.

When? Started over a year ago, I for ages thought I was my work's outsourced network IT dept (because they don't get much right) but my ISP went down a few weeks ago and using the nighthawk for a few days I spotted the connect 1st time every time.

Troubleshooting hurdle - My work device is a MacBook Pro running Big Sur (Catalina until last week) also and is heavily managed and thus installing Wireshark is not an option, I have to capture packets using pfSense thus can't compare the traffic to

Company firewall‍️
Likely to be a Cisco but I know it's exchanging OK port 443 TLSv1.2 and the cipher is EC Diffie-Hellman RSA_AES256_GCM_SHA384

Troubleshooting (= activity done and same result)
Firewall

• Changed the Outbound NAT rule generation between (AON - Advanced Outbound NAT), Automatic, Hybrid and even built a manual rule.
• Shutdown pfBlockerNG, unbound, DHCP etc etc and made the firewall as simple as possible.
• Even disable all packet filtering converting pfSense into a routing only platform!
• Rebuilt firewall on a standard image, no vlans etc and cable into firewall from the MacBook

Client

• Updated all drivers, firmware and software
• Rebuilt the MacBook
• Replaced (well upgraded) MBP 2019
• list itemTested with Big Sur or Catalina
• list itemTested with a work provided HP Windows 10 laptop
• Versions of the AnyConnect client
• cables replaced

Hardware
Firewall is A PC Engines APU2 with Intel NICs https://www.pcengines.ch/apu2.htm
Switches and AP's are UniFi

Thanks

Alex

[johnpoz]

I am on 2.4.5p1 and using cisco anyconnect (4.7.02036) as well - zero issues..

You do not have to sniff on your macbook - you can sniff right on pfsense, diag packet capture.

Looking in the message history on the anyconnect client on my work laptop.. Looks like I was connected from 11/13 through the 18th without any disconnects.

You think that maybe why it only connected for a a minute or so and then reconnected was an update it did, so it had to restart.

Example - here is a previous entry from the clients log.. Showing exactly that - it connected got an update of some kind and then reconnected..

 12:10:51 PM    Hostscan is performing system scan
 12:10:52 PM    Hostscan is performing software scan
 12:10:52 PM    Hostscan state idle
 12:10:53 PM    Hostscan is waiting for the next scan
 12:10:55 PM    Establishing VPN session...
 12:10:55 PM    The AnyConnect Downloader is performing update checks...
 12:10:55 PM    Checking for profile updates...
 12:10:56 PM    Downloading AnyConnect Management VPN Profile - 100%
 12:10:56 PM    Checking for customization updates...
 12:10:56 PM    Performing any required updates...
 12:10:56 PM    The AnyConnect Downloader updates have been completed.
 12:10:56 PM    Establishing VPN session...

That is from 11/9 in the log..

If your saying connected for hours at a time, sounds like your trying to find a nonisssue..

There is zero to do on pfsense for this to work, nothing to do in in outbound nat, nothing to do in any other settings.. This would work right out of the box.. Your chasing ghosts if you ask me..

There are many things that could cause a reconnection - blip on the internet, blip with your isp, blip on your works isp, issue on their end, timer for amount of time you can be connected, blip on your machine, etc. Did you run through the diagnostics in the any connect client.. And look to see anything in bundle it creates.. If your concerned something is wrong - create that bundle and send it to your IT dept..

How exactly are you sure its connecting to the same peer.. Where your connecting to could be a cluster, and issue with the one you connected too, so it reconnects to the same fqdn and just ends up talking to a different device in the server end, etc..

[Alex the firewall]

@johnpoz

Hello thanks for the reply, I'm sure the VPN peer is the same, it's the same IP in the logs..

For clarity the issue is not that I cannot connect it's that connecting to the work VPN takes up-to 4min due to the multiple reconnections BUT on a simple device such as an ISP provided modem/router, 4G nighthawk or hotspot'ing to the phone all is good it takes 40sec

Re-writing this section
Troubleshooting hurdle - My work device is a MacBook Pro running Big Sur (Catalina until last week) also and is heavily managed and thus installing Wireshark is not an option, I have to capture packets using pfSense thus can't compare the traffic going via the pfSence firewall to hotspot / cheap modem.

Thanks

Alex

[Alex the firewall]

@johnpoz

Re the AnyConnect DART not much progress there, I sent it to our IT dept (outsourced) who do nothing and then close the ticket on a weekend avoiding the 24h to reopen time limit.

Looking at the file myself, not 100% sure how to read them but the only message of interest is the dead peer detection BUT not always before the reconnect SO I tried another experiment.

I connected my NightHawk 4G modem to the Firewall and configured it as a second uplink, disconnected my ISP and re-ran the test, same issue.

The constant is the pfSence software and firewall hardware, Wi-Fi or Wired makes no difference, changing the ISP / Telco makes no difference, changing the laptop makes no difference.

Cheers

[johnpoz]

I don't even think it takes 40 seconds on my work laptop - and its old POS ;)

Just looked through my log on my client 29 seconds to connect..

I suggest you get to with your IT dept.. But there is nothing special with you creating a vpn.

Pfsense doesn't know packet A from packet B for what it is.. Its udp or tcp, and it passes it on and changes the ports for the NAT.

Are you doing something odd with scrub, or mss... I can tell you I have pfsense - and have ZERO issues maintaining a connection with anyconnect.. I show my current connection being up for 12 days +

Its not doing any static port nat, etc.

Are you running IPS that could be seeing something odd in the traffic and blocking it?

Love to help you - but it sure is not something wrong in pfsense.. With how many people are working from home, and any connect is a very common work thing.. I would think if there was something wrong the boards would be on fire..

I can not even think of anything you could turn on to cause the problem.. Other than messing with your mtu, or scrubbing, etc..

[stephenw10]

The biggest difference between pfSense and most soho style routers is that pfSense will randomise the source port of outgoing traffic by default.

You said you tried using different outbound NAT modes but did you actually set a static port rule for your client device?

I could imagine the remote side starts to connect and then rejects it based on an unexpected source port and has to fall back to some other mode or something similar.

Steve

[johnpoz]

That could be an issue sure... But this is exactly the reason I posted my states, where it shows the source port was changed.

And not sure what soho routers your looking at ;) But everyone I have seen does source port changing as well.. This is how napt works.. Its possible his doesn't do that? But I am not aware of cisco anyconnect caring.

If that was the case - what are the odds that the source port would end up the same after the nat.. Roughly 65k to 1 ;) I just don't see him ever connecting if that was the case..

His mention of dead peer detection.. I take it they are using DTLS then vs ipsec for their connection.. On the client you can see for sure under the stats tab .. For example mine is using IKEv2/IPsec NAT-T

You really need to get with your IT if your having issues maintaining or getting a connection.. If this was an issue with pfsense, the boards would be lit up with issues..

[stephenw10]

Mmm. I've never looked at an Anyconnect server but I imagine it has some configurable options.

Just from a high level when you see soho device X works fine and pfSense does not it's usually because of source port randomization.

And usually some crappy app that has been written assuming static ports. I do not expect Anyconnect to fall into that category though!

Steve

[johnpoz]

DTLS can be tricky - there are 2 tunnels that are brought up.. the normal TLS one which is control, and then the actual data tunnel which is just UDP..

If he is having connectivity issues, this can be problematic for sure. The whole point of the dead peer configuration..

Without both sides - knowing how its all configured.. etc.. It can be troublesome to troubleshoot what could be the problem. Which is why he really should get with his IT dept.. They have all the logs on their end, they can see the logs from his client (even if he has to send them via dart) etc..

[Alex the firewall]

@johnpoz

@johnpoz cheers for the help, it's appreciated..

Here's a packet dump and a screenshot. The host x.x.x.76.443 is the firewall and its a SSL VPN on port 443 as you can see the connection comes up and then resets ~1min and then stays connected.

Re How exactly are you sure its connecting to the same peer I can see the same fqdn and IP address in the packet dump.. I do take your point about blips on networks but 10/10 times its reconnecting after ~1min and then stays connected for up to the VPN limit of 15hours

I tried to post the packet capture but this site thinks it's spam?

Thanks

Alex

[Alex the firewall]

@johnpoz DTLS can be tricky - there are 2 tunnels that are brought up.. the normal TLS one which is control, and then the actual data tunnel which is just UDP..

Hang on, i'll BRB

[Alex the firewall]

@johnpoz

Yep, threw in an ip any any rule for a test!!

Yep, the port 443 UDP traffic, because I can't get a packet capture from the work laptop I couldn't see the 0 packet length and our company documents say SSL port 443 so I went with TCP and because it worked but then re-connected / failed back to TCP..

21:23:41.717625 IP 192.168.30.40.53444 > x.x.x.x.443: tcp 37
21:23:41.734610 IP x.x.x.x.443 > 192.168.30.40.53444: tcp 37
21:23:41.737181 IP 192.168.30.40.53444 > x.x.x.x.443: tcp 0
21:23:42.198962 IP 192.168.30.40.55546 > x.x.x.x.443: **UDP**, length 102
21:23:43.723197 IP 192.168.30.40.55546 > x.x.x.x.443: **UDP**, length 100
21:23:43.723234 IP 192.168.30.40.55546 > x.x.x.x.443: **UDP**, length 100
21:23:44.365881 IP 192.168.30.40.55546 > x.x.x.x.443: **UDP**, length 134

Thank you

[johnpoz]

@alex-the-firewall said in Cisco AnyConnect VPN behind a pfSense 2.4.5:

so I went with TCP

Meaning what? You altered the default any any rules? When you were sniffing you only did tcp? You made no mention of alerting the default lan rule which is any any..

I take it your working now? Or you still not coming up on UDP? And falling back to tcp?

You sure your IT dept has udp open on their end? I have seen it happen ;)

Your IT dept would of seen that right away if they bothered to look into it at all..

[Alex the firewall]

@johnpoz Hello and thanks

Yes I only had TCP port 443 outbound from my work VLAN and after adding UDP all is better. I'll VPN into work and update that wiki page