Posting to a forum issue

Pippin

Hi,

Having a strange issue posting to a forum.

Network looks like this:
Laptop <CAT5E > pfS <USBcable>Phone/4G
That works fine, no problems.

On Laptop I browse to a forum, posting there, all works fine.
When I start OpenVPN on the Laptop to a server on the net, tunneling through pfS, I can still browse that forum but trying to post a message just hangs showing "connecting to forum.url" in the browser until it times out.

When I take pfS out of the equation all works fine, with or without OpenVPN.

I tried en/disabling the hardware offloading, set it to conservative instead of normal, prefer IPv4, various combinations, looking at logs finding no clue.
There are no packages active.

Someone have an idea of what is going on?

Thanks.

stephenw10

I have seen similar odd behaviour like that when there were MTU issues on the connection. I think it was IPSec in that case but I could see it happening here.
Try setting in the custom options field mssfix 1300 on the OpenVPN client.

Steve

Pippin

Hi,

mssfix 1410 does the job, 1411 fails.

Then my question would become, why?
Because this is the first time I encounter this and would like to understand why when pfS is in the middle I need to mssfix it?

Thanks.

stephenw10

It's failing to detect a restriction somewhere between you and that remote host for some reason.

It was just that one host that was not working I assume?

https://en.wikipedia.org/wiki/Maximum_segment_size

Steve

Pippin

Hi,

Please define "It" in "It's failing".
I visit about ten forums and just one showed this behavior.
Never encountered this before, this Laptop has seen about 190 different networks, mix of wired and WiFi.

En/disabling scrub did not help either.

On to reading, thanks.

stephenw10

I would guess it's the TCP end points not correctly detecting the maximum segment size. Since you only see it with that site it could be a problem at that end. I'd have to go and refresh my memory on this stuff too though.

Steve

Pippin

Not had the opportunity to make captures but will do.

Although about IPv6, at the time I can only wonder if it's anyhow related to this (and other posts popping up):
https://forum.netgate.com/topic/138406/still-weird-issues-with-fragmented-ipv6-dns-packets-2-4-4-p1
The thing is, when pfS is removed from the path, it all works...

Thanks.

Pippin

Attached a capture on WAN interface.
The gateway for pfS is 192.168.42.129, that is Android USB tether interface.
pfS is 192.168.42.137

Destination unreachable, fragmentation needed, MTU of next hop 1410 comes back from .129
0_1544121910529_wanwithvpn.pcap

Next capture is on enp1s0 (192.168.51.2) of Laptop.
Same story, ICMP message arrives.
0_1544121931562_enp1s0vpn.pcapng

Next capture is on tun0 (10.8.8.80) of Laptop.
The only thing w.r.t. destinations that show up are the DNS root servers.
0_1544121949103_tun0withvpn.pcapng

Laptop routes all traffic through VPN.
Help in understanding what is going on is appreciated.

Thanks.

Edit:
Uploading files did not work, disabled NoScript and now works.
However, downloading capture from pfS results in a *.cap file which is not accepted by forum.
Renaming wanwithvpn.cap to wanwithvpn.pcap worked.

stephenw10

When you remove pfSense you;re running the OpenVPN client directly on the laptop? To the same endpoint?

We did see an issue with traffic coming across OpenVPN failing to pass packet fragments a while back. However that was only if the interface was unassigned or the rules passing the traffic were not on the assigned interface. Yours must be assigned here.

Steve

Pippin

The laptop is running the OpenVPN client and connecting to a remote server.
pfS is not involved in running OpenVPN, just sits in between and when removed it works.
Next time I will do capture on OPT1 too, Laptop is connected there.

Thanks.

stephenw10

Ah, OK I misunderstood. So pfSense is not handling the VPN termination in either case?

In that case it would appear to be limiting the route MTU in some way that OpenVPN does not detect?

Are you able to post to the forum if you are not running through the VPN but are using pfSense?

Steve

Pippin

Correct
I'm not sure but I think OpenVPN does not know about outside MTU after connection is established. Please see attached diagram and --mssfix max (and --fragment) in manual 2.4:
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
Yes

Thanks.

stephenw10

It does include '--mtu-disc'. FreeBSD doesn't support it but Linux does if your laptop is running that.

Hard to say why pfSense appears to be reducing the apparent MTU size though.

Steve

Pippin

Hi,

OpenVPN's --mtu related directives are "inside tunnel" options, AFAIK.

Also, the manual is not correct, --mtu-disc is not working IIRC.

Thanks.

Pippin

Made a capture on OPT1, if someone is willing to take a look at it.

Thanks.
0_1544569894586_opt1withvpn.pcap

Pippin

Now reading here:
https://github.com/OpenVPN/openvpn/blob/master/INSTALL

If run through a firewall using OpenBSDs packet filter PF and the
filter rules include a "scrub" directive, you may get problems talking
to Linux hosts over the tunnel, since the scrubbing will kill packets
sent from Linux hosts if they are fragmented. This is usually seen as
tunnels where small packets and pings get through but large packets
and "regular traffic" don't. To circumvent this, add "no-df" to
the scrub directive so that the packet filter will let fragments with
the "dont fragment"-flag set through anyway.>

Could this somehow be tested?

Thanks.

stephenw10

Hmm, Linux sends fragmented packets with df set? Seems.... interesting.

You can disable pfscrub entirely to test that. You could probably make a custom rules file and load it temporarily with no-df set.

I haven't see this for a long time but we have previously seen OpenVPN tunnels where fragmented packets would make it across the tunnel but never be sent out of the local interface at the other end. That only happened on OpenVPN tunnels where the interface was not assigned. Assigning it worked around it, somehow altering how pf handles the packets. I could see that being this issue.

Steve

JKnott

@stephenw10 said in Posting to a forum issue:

Hmm, Linux sends fragmented packets with df set? Seems.... interesting.

The DF flag tells routers not to fragment. The originating device is still free to fragment.

stephenw10

Mmm, I guess so.

JKnott

@stephenw10 said in Posting to a forum issue:

Mmm, I guess so.

If the source wants to send a packtet that's too big to fit within the MTU, it might split it into more than one piece. This is typically done with UDP, but not TCP. These days, the world is moving to path MTU detection. It's mandatory on IPv6 and now used on IPv4. On Linux, with IPv4, the DF flag is set on everything, but in Windows TCP only. As for TCP, don't forget that every packet is a fragment, unless the entire data can be carried in a single packet.

The reason for moving away from fragmentation is to reduce the performance hit at routers, along with reassembling at the destination.