22.05 - DCO and OpenVPN issue

swixo

@stephenw10 Yeah - for me all iOS and MacOS have issue.
My platform for pf is all Netgate 1537 appliances.

jimp

I have seen an occasional glitch on OSX but I'm not sure if it's specific to OSX. The first start after a reboot sometimes it won't ping, but if I restart the server and reconnect the client it can. But since it's so intermittent and doesn't happen every time (and didn't log anything different that I recall), I hadn't been able to nail down anything solid enough to call a problem yet.

If you can reproduce it reliably, check the ifconfig output for the interface, the contents of the routing table, and the OpenVPN logs for when it doesn't work, then restart the server and see if you can reconnect the client and pass traffic. If you can, then compare the logs and other output and see if you notice any differences.

swixo

@jimp said in 22.05 - DCO and OpenVPN issue:

If you can reproduce it reliably, check the ifconfig output for the interface, the contents of the routing table, and the OpenVPN logs for when it doesn't work, then restart the server and see if you can reconnect the client and pass traffic. If you can, then compare the logs and other output and see if you notice any differences.

Ok! I tested and I was able to get traffic to flow ONCE after I closed the tunnel, restarted the server and restarted the tunnel. But subsequent initiations of the tunnel resulted in no data flow.

This is the ifconfig for the interface - and it was the same in the fail and non fail condition:

utun10: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
	inet 192.168.41.2 --> 192.168.41.2 netmask 0xffffff00

I didn't see anything error like in the logs. Just the usual stuff.

s

jimp

That looks like the client side, what about differences on the server side when it works vs when it doesn't?

swixo

@jimp I have not been able to 'jiggle' it into working by disconnecting/restarting and reconnecting again. It just fails:

This is ifconfig from Server side for each state

tunnel disconnected:

ovpns2: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::3eec:efff:fe79:f1f2%ovpns2 prefixlen 64 tentative scopeid 0x19
	inet 192.168.41.1 --> 192.168.41.2 netmask 0xffffff00
	groups: openvpn
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Connected/Failing (No data pass)

ovpns2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::3eec:efff:fe79:f1f2%ovpns2 prefixlen 64 scopeid 0x19
	inet 192.168.41.1 --> 192.168.41.2 netmask 0xffffff00
	groups: openvpn
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Hope this helps.

jimp

I think this is what I hit yesterday with multiple clients failing (only the first client, e.g. .2 worked). We got a fix in yesterday afternoon and made a new build. You should be able to update to that and try it.

swixo

This post is deleted!

swixo

@jimp I may have spoken too soon -- It worked once - but shortly after - same problem returned.

Did a server restart - and its blocking data again w/DCO. Sorry for any false start - just reporting what I'm seeing.

jimp

Odd. I can't reproduce any problem now, and I could before. I've restarted the server and clients multiple times. Not just reboots but daemon restarts.

swixo

@jimp So I tried again with MacOS - working with DCO. No erros in logs. Connected with iOS - nothing. And as soon as I connected with iOS, the MacOS connection stopped working too.

Log errors at time of failure:

Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 TLS Error: tls-crypt unwrapping failed from [AF_INET]192.168.1.212:54428 (via [AF_INET]192.168.1.1%)
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: packet replay
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: bad packet ID (may be a replay): [ #9 / time = (1655315314) 2022-06-15 10:48:34 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 TLS Error: tls-crypt unwrapping failed from [AF_INET]192.168.1.212:54428 (via [AF_INET]192.168.1.1%)
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: packet replay
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: bad packet ID (may be a replay): [ #8 / time = (1655315314) 2022-06-15 10:48:34 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

maverick_slo

For me, Android now works OK.
But I had to disable DCO, because it`s like 40% slower than without it...

jimp

@maverick_slo said in 22.05 - DCO and OpenVPN issue:

For me, Android now works OK.
But I had to disable DCO, because it`s like 40% slower than without it...

If your hardware supports AES-NI, make sure the AES-NI kernel module is loaded (System > Advanced, Misc tab).

The only time we've seen DCO be slower is when the hardware supports crypto acceleration but it's not enabled in the kernel/modules. DCO is in the kernel so it can't just latch onto AES-NI via OpenSSL like non-DCO OpenVPN can.

jimp

@swixo said in 22.05 - DCO and OpenVPN issue:

@jimp So I tried again with MacOS - working with DCO. No erros in logs. Connected with iOS - nothing. And as soon as I connected with iOS, the MacOS connection stopped working too.

Log errors at time of failure:

Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 TLS Error: tls-crypt unwrapping failed from [AF_INET]192.168.1.212:54428 (via [AF_INET]192.168.1.1%)
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: packet replay
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: bad packet ID (may be a replay): [ #9 / time = (1655315314) 2022-06-15 10:48:34 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 TLS Error: tls-crypt unwrapping failed from [AF_INET]192.168.1.212:54428 (via [AF_INET]192.168.1.1%)
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: packet replay
Jun 15 10:48:35	openvpn	2705	192.168.1.212:54428 tls-crypt unwrap error: bad packet ID (may be a replay): [ #8 / time = (1655315314) 2022-06-15 10:48:34 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

I still can't reproduce any problems here. Replay errors suggest it's a network-level problem with packets arriving out of order, though.

swixo

@jimp Updated to 22.05.r.20220617.0613 this AM.

Similar situation - First client to connect - its good. Second client to connect - Both lose data flow.

maverick_slo

It7s loaded...
AES-NI CPU Crypto: Yes (active)

But still it is really slower...

tman222

Hi @jimp - just upgraded my 22.01 system to 22.05 RC and seeing the same issues that @swixo and @maverick_slo have described when DCO is enabled on OpenVPN.

In my case I'm using two clients:

iPhone with iOS 15.5 and version 3.2.3 of the OpenVPN Connect client for iOS
Macbook Pro with macOS 12.4 and latest version of Tunnelblick VPN client (3.8.7a)

Each client will work fine when connected individually. If I connect both then both will lose data flow as soon as the second client is connected (the order in which they are connected doesn't seem to matter). I've increased the verbosity on the OpenVPN logs but can't really see anything that looks to the a problem. Are there any settings on the client or server side I should doublecheck? I basically just enabled DCO on the server and tried things out. As a sanity check, I did disable DCO and then connected both clients - everything worked fine (i.e. data flow was not interrupted).

EDIT: Just tried with a non Apple client as well (Debian 11.3 Linux system). As soon as second client connects both lose data flow.

Regarding speed:

On the iPhone (WiFi) having DCO enabled is slower for me also (especially download) compared to no DCO with Fast I/O and 512KB buffer sizes.
On the Macbook Pro (wired) it is just as fast if not a tad faster, which I thought was interesting.

Thanks in advance for your help.

tman222

One additional thing I thought of: I did create an OpenVPN interface ovpns1 when I originally set things up. Are there any configuration settings I should be checking with respect to that interface (e.g. MTU, MSS, etc.)? Thanks again.

tman222

I'm curious if you guys had any idea on how to troubleshoot this further? The only thing I have not tried yet is creating a brand new OpenVPN server (tunnel) from scratch to see if that resolves the data flow issues. @swixo and @maverick_slo - have you guys tried this yet? Thanks in advance.

swixo

@tman222 I have not tried deleting the tunnel and starting over - although I may create a new one and see if it is ok.

We have a bug in either case and I want to preserve the failure condition so I can verify a proper fix if offered.

swixo

@swixo This issue has been reliably reproduced (and still failing) on 22.05-RELEASE (amd64).