OpenVPN TAP pfSense Gateway Website Inaccessible

iorx

Also has a TAP question in here, I must smell funny too because No replies . I was though nowhere as detailed as you here, so maybe that explains the lack of response in my case.

Have you tried manipulating MTU size?

I have a bit of limited network experience at just this scenario, as I struggle myself to get a simple Server-Client (Windows 10) bridge work as intended.
But to me this case could be about fragmentation I believe.

JKnott

@iorx said in OpenVPN TAP pfSense Gateway Website Inaccessible:

Have you tried manipulating MTU size?
I

I suspect you're not getting responses as most (all?) people use TUN mode.

BTW, if you use TAP, your MTU must be the same on the tunnel and both LANs.

seejay

@iorx Default MTUs are set on all interface cards at both pfsense gateways, so there should be no mismatch and the default 1500 should be getting used.

seejay

I will occasionally bump this in the hopes either an insight is shared or I figure it out first.

JKnott

@seejay said in OpenVPN TAP pfSense Gateway Website Inaccessible:

Default MTUs are set on all interface cards at both pfsense gateways, so there should be no mismatch and the default 1500 should be getting used.

If your WAN connection has a 1500 byte MTU, your OpenVPN MTU will be 1472, due to 28 bytes lost to headers.

seejay

@JKnott Thanks. Are you suggesting I need to manually set MTUs on the WAN / LAN / OpenVPN interfaces and that defaults would cause this type of condition? If I set 1500 manually on the WAN, do I set 1472 on the OpenVPN adapter? What about the bridged LAN adapter?

seejay

I also see this entry from initialization, maybe the MTU config getting set?

/usr/local/sbin/ovpn-linkup ovpnc1 1500 1605 init

seejay

Confirmed with ping -n 1 -l 1472 -f [router ip] the max MTU

seejay

set WAN, LAN, and OPENVPN adapter in interface config all to 1500 on both sides of the tunnel. No impact.

seejay

Still see this in OpenVPN logs:

Nov 28 09:43:48 openvpn 23207 Local Options String (VER=V4): 'V4,dev-type tap,link-mtu 1605,tun-mtu 1532,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA256,keysize 256,secret'

Doesn't seem right.

JKnott

@seejay said in OpenVPN TAP pfSense Gateway Website Inaccessible:

@JKnott Thanks. Are you suggesting I need to manually set MTUs on the WAN / LAN / OpenVPN interfaces and that defaults would cause this type of condition? If I set 1500 manually on the WAN, do I set 1472 on the OpenVPN adapter? What about the bridged LAN adapter?

If your WAN is 1500, the VPN will be 1472. If the LAN at either end is other than 1472, then you will have an issue with handling packets that have the full MTU. A TAP VPN is functionally the same as a bridge or switch, in that all connected networks must support the same MTU. With IP, the 2 end points negotiate the MTU, based on the end with the smallest MTU. The VPN MTU will not be considered in that process. If it was a TUN VPN, then the smaller MTU will cause the router to fragment the packet or send a too large ICMP message back to the source. That cannot happen with a TAP VPN, so there is no mechanism to reduce the MTU.

Bottom line, the WAN MTU determines the VPN MTU, which in turn determines the LAN MTUs.

seejay

@JKnott So if I force both WANs to 1500 MTU, and both LANS to 1472 MTU, do I need to apply an mssfix or other value to the openvpn config or will it determine it correctly? I separately dont understand how when setting WAN1500, LAN1472, OpenVPN server reports on startup:

Nov 28 10:04:51 openvpn 82520 Local Options String (VER=V4): 'V4,dev-type tap,link-mtu 1605,tun-mtu 1532,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA256,keysize 256,secret'

Nov 28 10:04:51 openvpn 82520 Expected Remote Options String (VER=V4): 'V4,dev-type tap,link-mtu 1605,tun-mtu 1532,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA256,keysize 256,secret'

Lastly, I would expect this problem only to occur on packets large enough in size to approach this boundary. The website packets for the gateway that fail are usually no greater than a couple 100.

seejay

A quick LAN speed test across the VPN shows better read performance undoubtedly forcing WAN at both ends to 1500, LAN at both ends to 1472, and mssfix on both OpenVPN configs to 1432. sadly, the respective pfsense websites at opposite ends of the bridge still are not coming up (but still ping).

seejay

Taking the MTU advice I applied it back to our dedicated NIC configuration (Separate set of NICs on the pfsense boxes attached to the same LAN but with no IP assigned in pfsense). Seems to be stable so far with no dropouts, and I can again reach the pfsense websites across the bridge. Going to run this way for a bit and see if we stay stable with 0% packet loss.

JKnott

@seejay said in OpenVPN TAP pfSense Gateway Website Inaccessible:

So if I force both WANs to 1500 MTU, and both LANS to 1472 MTU, do I need to apply an mssfix or other value to the openvpn config or will it determine it correctly

It should be OK, as there will not be a conflict with MTU size.

seejay

Back to square zero. Separate dedicated NICs with no IP observe packet loss and instability, using same LAN nic as PFSENSE LAN IP assignment makes everything stable but cant access web pages across the bridge. MTUs set properly now in both cases.

coffeecup25

@seejay Are the rules out of order on your firewall. For example, is the 'restrict everything else' above the entry that says your OpenVPN connection is OK? (Voice of experience talking)

Also, is the lan to use on the TAP server configuration the same as the one you want to access? (Sorry, but I did not read all the details about all the problems.)

Finally, I used this guide to set up my TAP configuration (https://hardforum.com/threads/pfsense-2-0-1-openvpn-configuration-guide.1663797/). Any clues here?

(I have 3 configuration 1 TUN pass through only, 1 TUN with full access, 1 TAP with full access. Why: because I wanted to)

seejay

@coffeecup25 said in OpenVPN TAP pfSense Gateway Website Inaccessible:

@seejay Are the rules out of order on your firewall. For example, is the 'restrict everything else' above the entry that says your OpenVPN connection is OK? (Voice of experience talking)
Also, is the lan to use on the TAP server configuration the same as the one you want to access? (Sorry, but I did not read all the details about all the problems.)
Finally, I used this guide to set up my TAP configuration (https://hardforum.com/threads/pfsense-2-0-1-openvpn-configuration-guide.1663797/). Any clues here?
(I have 3 configuration 1 TUN pass through only, 1 TUN with full access, 1 TAP with full access. Why: because I wanted to)

Thanks for the response. In order of your questions:

@seejay Are the rules out of order on your firewall. For example, is the 'restrict everything else' above the entry that says your OpenVPN connection is OK? (Voice of experience talking)
-- Yes, the anti-lockout rule is always the first entry, and all IPV4* traffic is set to pass on the OpenVPN interface.
Is the lan to use on the TAP server configuration the same as the one you want to access? (Sorry, but I did not read all the details about all the problems.)
-- Definitely. I've built these bridges multiple times, and given I can access all of the other resources on the LAN normally I am not concerned that I've confused this.
Finally, I used this guide to set up my TAP configuration (https://hardforum.com/threads/pfsense-2-0-1-openvpn-configuration-guide.1663797/). Any clues here?
-- Sadly this looks more like a server/client setup for TAP as opposed to site-to-site configuration, but similar concepts.

Ultimately no matter which TAP/bridging configuration I've employed for site-to-site TAP I have odd issues like the one outlined in this post, or random packet loss and/or TCP resets. You've seen me go through things like the MTU and other diagnosis ad nauseum to no avail.

For comparison I've also set up a tunnel configuration for the same network(s) and its far superior. Of course it doesn't fully meet the original use case but I'm trying out some other tooling to try and work around the lack of broadcast traffic out of the box. Ultimately unless I'm missing something there seems to be a bug in the implementation of the bridged OpenVPN interface (or something overlooked non obvious in the default setup).

Rai80

I've been running a TAP openvpn tunnel between two sites without any issues. I use tap for broadcast/multicast traffic, which is not possible with a tun openvpn tunnel.

Check the following:

net.link.bridge.pfil_bridge = 0 to disable filtering on bridge
net.link.bridge.pfil_member = 0 to disable filtering on bridge-member
Assign en IP address to bridge instead of a bridge-member interface

seejay

New suggestions, will have to give a try on #3. On one hand I can see that being the issue, on the other hand according to:

https://docs.netgate.com/pfsense/en/latest/interfaces/interface-bridges.html

the IP address assignment should be acceptable in either place (the bridge with no members, or one member only in the bridge).