Strange MTU issues

[raptor01uk]

Hi Everyone,

Been using Pfsense for a number of years so I know my way around (or so I thought). I'm running Pfsense as a vm on a KVM host and already ran into the issues with virto network drivers. I've disabled checksum offloading and it seems to have resolved that problem.

I now have a strange MTU issue.

Setup:

Two KVM servers
Two Pfsense boxes with HA (VRRP) - One Firewall on each KVM instance. Both running 2.3.2-RELEASE-p1
Four interfaces on each box

1. WAN
2. LAN
3. PFSYNC
4. Server DMZ

Situation:

I am in control of the network surrounding the servers and have a test line into the environment. The pfsense WAN and DMZ are public IP addresses and NAT is disabled to the WAN from the DMZ.

The problem:

Using MTURoute I get 1500 bytes to the WAN VRRP and to the each FW static IP address. MTURoute to the Server DMZ IP address or VRRP or any server in the DMZ and I get 1500 all the way to the router in front of the KVM boxes and then 1496 bytes to the pfsense Server DMZ IP address range.

All MTU settings on the PFsense boxes are default (1500) and the KVM host has a layer 2 MTU of 9000 bytes.

MTURoute - FW WAN VRRP

C:\Users\Raptor\Desktop>mturoute.exe -t (You can't see this)
mturoute to "Firewall WAN VRRP", 30 hops max, variable sized packets

• ICMP Fragmentation is not permitted. *
• Speed optimization is enabled. *
• Maximum payload is 10000 bytes. *
  1  +-  host: 192.168.5.254        max: 1500 bytes
  2  +-  host: Test line GW           max: 1500 bytes
  3  +-  host: Core           max: 1500 bytes
  4  +-  host: GW Router   max: 1500 bytes
  5  +-  host: Firewall WAN VRRP  max: 1500 bytes

MTURoute - Firewall DMZ VRRP

C:\Users\Raptor\Desktop>mturoute.exe -t (You can't see this)
mturoute to "Firewall DMZ VRRP, 30 hops max, variable sized packets

• ICMP Fragmentation is not permitted. *
• Speed optimization is enabled. *
• Maximum payload is 10000 bytes. *
  1  +-  host: 192.168.5.254                        max: 1500 bytes
  2  +-  host: Test Line GW                        max: 1500 bytes
  3  +-  host: Core                                   max: 1500 bytes
  4  +-  host: GW Router                            max: 1500 bytes
  5  …-+++++++++.-++  host: Firewall DMZ VRRP  max: 1496 bytes

MTURouter - Server within the DMZ

C:\Users\Raptor\Desktop>mturoute.exe -t (You can't see this)
mturoute to "Server within DMZ", 30 hops max, variable sized packets

• ICMP Fragmentation is not permitted. *
• Speed optimization is enabled. *
• Maximum payload is 10000 bytes. *
  1  +-  host: 192.168.5.254                          max: 1500 bytes
  2  +-  host: Test Line GW                          max: 1500 bytes
  3  +-  host: Core                                     max: 1500 bytes
  4  +-  host: GW Router                              max: 1500 bytes
  5  ...-+++++++++.-++  host: Server within DMZ  max: 1496 bytes

My theory:

I think that KVM is adding a 4 byte vlan tag but I can't be sure, however this would explain the loss of only 4 bytes. But I get this issue when going to the Pfsense IP address so it would suggest that its not dropping back onto the KVM virtual NIC so just passing through Pfsense and back out.

Anyone had this issue before or have a similar setup? I would be grateful for anyone's input.

Thanks in advance.

[Harvy66]

QinQ tags so happen to be 4 bytes

https://en.wikipedia.org/wiki/IEEE_802.1ad

[raptor01uk]

That's another possible explanation.

Am I right in saying then that the virtual KVM NIC uses QinQ to encapsulate traffic? There are not currently any Vlans configured within Pfsense so it just sees them as access ports however there is a trunked 4G etherchannel supplying the servers so if KVM uses QinQ then it might add 4 bytes for the Vlan tag then another 4 bytes for the QinQ tag.

Thanks for the suggestion.

[raptor01uk]

Nope wasn't QinQ tags.

Seems to be only traffic that passes through Pfsense. If you just test to either side it's fine but the moment you test through in either direction it gets this issue.

[raptor01uk]

Worked out this issue in the end.

It was the kernel drivers for the server nics.

Thanks for your help everyone.

[VegO]

Hi, I've faced the same issue, just now.
I run couple psSense 2.4.4-RELEASE as a virtual machine on a XenServer 7.2.0 HA cluster without any issues about 2 years.
Before now, I used dedicated virtual network interfaces to connect virtualized pfSense VMs to my VLANs. Couple days ago I decided that its time to connect pfSence to a trunk and configure VLANs on it.

I did it and it works but with MTU issues. It looks like it works but at the same time, heavy traffic like SMTP over TLS doesn't go through or heavy https web pages can't be opened. 1469 is the largest ICPM packet size can go through the VLAN interface.

So, I would appreciate if anyone can help to resolve the issue and a specially if the topic starter can share his experience and explain what was wrong with the kernel drivers for the server NICs.