22.05 - DCO and OpenVPN issue

tman222

I'm curious if you guys had any idea on how to troubleshoot this further? The only thing I have not tried yet is creating a brand new OpenVPN server (tunnel) from scratch to see if that resolves the data flow issues. @swixo and @maverick_slo - have you guys tried this yet? Thanks in advance.

swixo

@tman222 I have not tried deleting the tunnel and starting over - although I may create a new one and see if it is ok.

We have a bug in either case and I want to preserve the failure condition so I can verify a proper fix if offered.

swixo

@swixo This issue has been reliably reproduced (and still failing) on 22.05-RELEASE (amd64).

jimp

There must be something different about your setup. I still can't reproduce a problem here. I have three clients connected (pfSense, Windows, and OSX) and they all have working connectivity across the VPN.

swixo

@jimp I must have a setting aggravating it - user tman seems to have same issue.
Just in case - here are my settings:

DCO=on
Mode: TUN
Proto: UDP/v4
interface: Any 
Use Tls Key: Checked
TLS Usage: Enc and Auth
TLS KeyDir: Both
No Cert Rev List
No OCSP Check
DH: 4096
ECDH: secp384r1
Algorithms: AES-256-GCM
Fallback: AES-256-GCM
Auth Digest: SHA256 (256)
HW Crypto: Intel RDRAND
Cert Depth (Client+Server)
Client Certificate Key Usage: Enforce Checked
Tunnel Network /24
no IPV6
Redirect: Force all IPv4 Thru tunnel: Checked
No Redirect IPv6 checked
Concurrent connections: blank
Compression: Refuse non stub
Inter Client Comm: Checked
Duplicate Conn: Checked
Dup Conn Limit: Blank
Client Dynamic IP: Checked
Topology: One IP address per client common subnet
Ping: Inactive: 0
Method: Keepalive
interval: 10
Timeout: 60
DNS Default Domain: Checked
DNS Server Enable: Checked
Block Outside DNS: Unchecked
Force DNS Update: Unchecked
NTP Server: Checked
Custom Options: push "route 1.2.3.4 255.255.255.0"
UDP Fast I/O: Unchecked
Exit Notify: Disabled
Send/Receive Buffer: Default
Gateway Creation ipv4 only

jimp

I'm using RA SSL/TLS but that shouldn't matter.

TLS Usage: Enc and Auth

Mine: Auth only

TLS KeyDir: Both

Mine: Default

DH: 4096

Mine: 2048

ECDH: secp384r1

Mine: Default

HW Crypto: Intel RDRAND

Set that to "No", you aren't going to gain anything from that one, especailly with DCO.

Redirect: Force all IPv4 Thru tunnel: Checked

I don't have this set but it's probably not going to change anything.

Inter Client Comm: Checked

Mine: Unchecked

Duplicate Conn: Checked

Mine: Unchecked (each client has a unique cert)

Client Dynamic IP: Checked

Mine: Unchecked

DNS Default Domain: Checked

Mine: Unchecked

DNS Server Enable: Checked

Mine: Unchecked

NTP Server: Checked

Mine: Unchecked

Custom Options: push "route 1.2.3.4 255.255.255.0"

Mine: Blank (but that one route alone should be fine)

A lot of those, like the pushed DNS options, shouldn't make a difference. I'd focus on trying to change one thing at a time to see if it helps and if it doesn't, change it back and try the next one. That way you can isolate the potential problem there.

I'd try changing TLS auth first to auth only, I'm not sure if anyone has tried TLS enc+auth with DCO yet.

Also you listed several options that should be hidden when you have DCO on (like inactive and UDP fast i/o), are those really showing in your GUI with DCO checked or did you transcribe those from somewhere else?

swixo

@jimp said in 22.05 - DCO and OpenVPN issue:

HW Crypto: Intel RDRAND

Set that to "No", you aren't going to gain anything from that one, especailly with DCO.

I'd try changing TLS auth first to auth only, I'm not sure if anyone has tried TLS enc+auth with DCO yet.

Also you listed several options that should be hidden when you have DCO on (like inactive and UDP fast i/o), are those really showing in your GUI with DCO checked or did you transcribe those from somewhere else?

My bad on that - I had flipped DCO off just to get the VPN going again and hadn't thought the UX would change the options.

I will test the changes you suggest and report.

swixo

@swixo Tried a bunch of these changes and made things worse.

Intel Hardware accel had no effect. But the Key Dir and Auth Only/Auth+Encry setting made a lot of difference. Had to export new profiles after every change to make sure clients were in sync.

Most of the time - I couldn't connect at all - or data never flowed, until turning DCO off - then it usually worked fine after that.

JeGr

@jimp Just to chime in:

we were rolling out 2 new boxes 6100/4100 to a customer and I set up OpenVPN RAS pretty default with new DCO setting.

Exactly same problem: .2 client can connect and route/transfer data, all other client IPs don't get ANY data at all sent through the connection. Switching off DCO immediatly works again. No spiffy or special stuff configured, just plain dead simple RAS setup with a single LAN network that gets pushed to the clients.

Clients are 2x windows boxes with win10, newest OVPN Client 2.5.9/x64 and have no problems whatsoever. Routes are just fine, traffic simply refuses to flow through the server if you're not client #.2 :)

System is on 22.05 stable

Cheers
\jens

stephenw10

Hmm, were those supplied with 22.05 or upgraded from 22.01?

Does that include traffic between IPs in the tunnel subnet directly?

JeGr

@stephenw10 Those were freshly installed with 22.01 and clean upgraded to 22.05 - at least there were no errors or other hiccups in the logs or anywhere to see.

Besides that it's a simple straightforward RAS style setup:

SSL/TLS + User Auth
DCO
tun L3
UDP/1194
TLS Key with TLS Auth (not auth+enc), default direction
VPN CA, VPN Cert, VPN CRL created
ECDH only
prime256v1
SHA256
no HW crypt (but AES-NI enabled kernel module)
cert depth 1 (C+S)
Strict User-CN Matching
Enforce Key usage
IP4 tunnel network 192.168.45.0/26 (to leave space to add another VPNs server later with .45.64/26, .45.128/26, etc.)
IP4 local network 192.168.40.0/24 (LAN)
compression: refuse any non stub (most secure)
dynamic IP selected
subnet
keepalive 5 30
DNS default domain set up to the locally used domain
DNS server set up to the local MS AD server
Gateway v4 only
Verb 3

nothing else set. The RAS clients aren't supposed to talk with each other so no, inter-client comm isn't a thing here :)

Cheers
\jens

stephenw10

I mean can clients other than .2 ping the server tunnel IP?

stephenw10

I've been unable to replicate this so far. Did you test disabling AES-NI?

JeGr

@stephenw10 said in 22.05 - DCO and OpenVPN issue:

I mean can clients other than .2 ping the server tunnel IP?

Ah that's what you meant! I just switched DCO back on and tested for you:

VPN net: 192.168.45.0/24
LAN net: 192.168.40.0/24

When DCO is on:

Client 1 connected as .45.2:
- ping to .45.1 (VPN GW) -> works
- ping to 40.1 (FW IP in LAN) -> works
- ping to 40.x (any other IP then Firewall) -> works
- can connect to e.g. Server on 192.168.40.10
Client 2 connected as .45.3:
- ping to .45.1 (VPN GW) -> works
- ping to 40.1 (FW IP in LAN) -> works
- ping to 40.x (any other IP then Firewall) -> DOESN'T work
- no connect to any other device on the LAN is working

As there are IPsec tunnels currently in use I couldn't disable crypto but it is set to QAT - not AES-NI - as it's a 6100 :)

Cheers

stephenw10

Ah, OK I was testing to the LAN IP. Retesting....

stephenw10

Ok, replicated it. Let me see if I can narrow it down....

swixo

@stephenw10 Whew! Glad this was tracked down.

stephenw10

OK, this looks like an internal routing issue. As a workaround applying outbound NAT to traffic leaving the LAN appears to allow it to route replies as expected. If you want to test DCO that is.

Steve

JeGr

@stephenw10 Would that be subject to a patch via "System Patches" or is the routing issue deeper than the patch system can go and requires a new build/version of some files? Just asking if that'd be hotfix'able.

stephenw10

It's probably not something that can be fixed with a run-time patch unfortunately. It looks to be in OpenVPN so something in the binary.

Steve