TLS Handshake failed fragmentation issue?

cannyit

Dear community I have following issue making me cracy sinze days:

My Setup:
Debian server with static pubic IP running wireguard + NAT via iptables.
I configured two peers (identically beside IP and keys)

When I connect from a windows client via wireguard everything works fine.
When I connect from pfsense via wireguard some connection failes.

I already checked it down to traces on the debian box.

During handshake server sends a "Server Hello" (with length > 1420) which should be nated into wireguard subnet.
With the windows client its fine and "Hello" is nated and answered.
But with the pfsense the Hello packet is not nated but instead aswered with

ICMP Destination unreachable (Fragmentation needed), MTU of next hop: 1420

on the public IP.

I checked several times configuration. Both connecttions using the same wireguard interface with identical settings. The server I try to reach is also the same. And I used the same client.

On the Debian server I just have ip forwading enabled and iptables with SNAT from Wiregurad network to public IP + masquerade.

It would be great to probably get some inputs and ideas what I could check or if someone facing simlia issue.

cannyit

I am quite sure that it is a kind of MTU and/or fragmentation issue. Because relevant packets are bigger as tunnel MTU 1420 (and bigger as ethernet 1500 MTU).
Strange is, that connection to wireguard windows client is not creating any ICMP fragmentaion packets but connection to pfsense does.
I was already thinking and testing with hardware offloading as a cause and tried disabling and enabling on both ends without success.
Also no luck with different MTU settings. For now network on both ends has MTU 1500 and tunnel is set to 1420 on both ends. Ping works fine in both directions. Other connections are also working in both directions except this strange behavour for lage packets.

cannyit

After really long try and error I found the cause of all the mess. It is in this case mandatory to enter a MSS into the wireguard interface. I thought its calculated automatic based on MTU so 1420 - 40 = 1380. but it is not!
After entering the MSS (1380) the connection is working like charm, also with large packets and hardware offloading.