Issue accessing certain websites

russm

Having an issue accessing certain websites when using PFSense as a router (disabled outbound NAT and it's sitting behind a UTM). The majority of the sites that we access come up fine. However I'm seeing more issues with microsoft hosted sites and yahoo. If I plug in directly to the UTM, the websites open right up. One particular website is bing.com. I've attached a cap from the PFSense. My local ip address is 10.0.200.220 and the website is the 13.107.21.200. packetcapture (2).cap

johnpoz

@russm said in Issue accessing certain websites:

I've attached a cap from the PFSense

I don't see this attached.

russm

Sorry.. thought I could just upload it. Says that it's an invalid image file.
https://drive.google.com/file/d/118Zwb5az8wl4txQJNfoHYqW4bh1JWEM8/view?usp=sharing

johnpoz

@russm said in Issue accessing certain websites:

https://drive.google.com/file/d/118Zwb5az8wl4txQJNfoHYqW4bh1JWEM8/view?usp=sharing

what is this 192 box?

Se the message about fragmentation needed.

russm

That is the WAN interface of the pfsense box. I did see that. Not sure what it means though.

johnpoz

@russm do you have your mtu set to something odd on your lan side?.. a 1514 wouldn't require frag..

russm

@johnpoz MTU is not set on any of my interfaces.

johnpoz

@russm I would double check that they are 1500 (default standard)

example

edit: could you check your settings here

Under Advanced networking.

russm

@johnpoz

xn0.1010: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496
	description: VLAN1010
	ether 7a:5d:9d:11:f0:43
	inet6 fe80::785d:9dff:fe11:f043%xn0.1010 prefixlen 64 scopeid 0x14
	inet 10.0.200.254 netmask 0xffffff00 broadcast 10.0.200.255
	groups: vlan
	vlan: 1010 vlanpcp: 0 parent interface: xn0
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

//edit//

johnpoz

@russm well that is odd...

Those are couple of my vlan interfaces. Also see my edit about segmentation offloading.. above.

edit: Hmmm not sure why it would lower your mtu on your vlan? Lets call in @stephenw10 he is the hardware guru around here - maybe he might know why that could/would happen.

russm

@johnpoz I did.. and edited my post. :)

johnpoz

@russm yeah I saw - I tagged Steve, he would be the guy I would ask about how that could or would happen, etc.

johnpoz

@russm I found this

https://forums.freebsd.org/threads/vlan-lagg-mtu-1496-expected.55313/

This is in Steve's wheelhouse for sure I would think... Lets give him a bit to see if he is online, if not next time I chat with him will for sure bring this thread to his attention. If I was having such an issue, he is the guy I would ask for sure ;)

I have not seen such an issue before. But that could explain the icmp message.

edit: what specific are you running pfsense on, what version? You running this on Xen hypervisor? I think xn0 is from that is it not?

russm

@johnpoz Thanks. I am running PFsense on XCP-NG. I just checked the MTU in there, and they're all set to 1500.

johnpoz

@russm I found this really OLD redmine that seems related, and shows the 1496

https://redmine.pfsense.org/issues/4103

Wish I could provide more help - but I have like zero experience with Zen.. Personally have never seen such an issue in like 30 years that I can recall.. But yeah that mtu is most likely related to your problem, I can say that for sure ;)

russm

@johnpoz Thanks for the help so far. Hopefully Steve can provide some insight.

kiokoman

@russm

is it relevant to your situation? ->

In a XEN Virtual machine configuration the XEN VM network interface is mapped to a bridge device on the XEN Host. The Bridge on the XEN host is mapped to a physical Ethernet interface.

If the user configures a VLAN interface on a bridge device it does not handle the four byte tag and the maximum frame size with the four byte VLAN tag data is still 1514 bytes. This will force the maximum MTU to be 1496 bytes, not 1500, on the VLAN interface.

If the user is using TCP or some other network protocol that negotiates a segment size then the 1496 versus 1500 MTU difference is not an issue. The side with the 1496 MTU will simply negotiate a smaller segment size. But if the network application is perhaps a multicast or some other protocol that does not negotiate an end to end segment size then the transmit from a MTU 1500 host to a host with an interface set to a MTU of 1496 will most likely result in the packet from the MTU 1500 host being dropped at the destination interface with the MTU 1496.

Details
So with the bridge being in between the XEN VM and the XEN Host how does one configure a VLAN for a XEN VM?

Create the VLAN interface on the XEN host and map a bridge to this interface. Then a XEN VM interface mapped to that bridge would automatically use the VLAN, one do not need to configure the VLAN 802.1q driver in the XEN VM. The XEN host VLAN interface takes care of adding and removing the VLAN tag as needed. The VLAN on the XEN host interface takes care of the four byte tag automatically so the user does not need to adjust the MTU size.

Keep the XEN host bridge device at a MTU of 1500 so it is compatible with the MTU of other systems in its subnet. Then create the VLAN under the XEN VM, with the 802.1q driver, and set its MTU to 1496. This will work OK for any network protocol that negotiates an end to end segment size. For network protocols that do not negotiate a segment size then any packet sent from a MTU 1500 host will get dropped at the XEN host bridge and never arrive at the XEN VM VLAN interface.

Set the XEN Host interface and matching bridge to a MTU of 1504. Then when the user creates the XEN VM VLAN it can use a MTU of 1500 and the additional 4 bytes added to the XEN HOST bridge and interface will allow the 1518 frame size. But this will place a MTU mismatch on the XEN HOST. Since a MTU of 1504 is an odd size not all network drivers will accept this MTU size.

Option 1 is the easiest to use and the MTU mismatches do not exist. But if there is a network design reason to only use the 802.1q driver under the XEN VM, then choice 2 or 3 can be used as long as the MTU mismatch drawbacks are considered.

johnpoz

@kiokoman Yeah I would think this is exactly what he is running into.

My take on that would be to do the tags on the actual host, and the pfsense should just have standard vm interfaces and not know about the tags..

russm

@kiokoman @johnpoz

Thanks. I am testing this now. I did notice that all of the interfaces that were passed from XCP-NG were set to 1500, while the VLANs were all lower, no matter which interface they were from.

russm

Initial tests are good. I'll reconfigure the VLANs this way tonight.