pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Mmm, because we also saws the vlan0 tags when we removed the access point and just used the switch.
That implies it's either not the AP or the switch or that it's both devices doing it. Which seems unlikely!

I got news. @stephenw10 and @johnpoz
I installed pfSense 22.05 on the another mini PC that has 2 NICS that uses igb and em drivers.

This is some info from ifconfig:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0 prefixlen 64 scopeid 0x1
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:54
        inet6 fe80::82ee:73ff:febb:e54%em0 prefixlen 64 scopeid 0x2
        inet ************* netmask 0xfffff800 broadcast ***************
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        inet 10.10.10.21 netmask 0xffffffff
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
igb0.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::82ee:73ff:febb:e55%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 59956

Now a tcpdump from pfSense igb0 - the new LAN interface with NATIVE, VLAN20 and VLAN30
pfsense_lan_tcpdump_new_machine.txt

Everything works, every WLAN, every VLAN, any host. No problem whatsoever.

Conclusions please guys

Tell me @stephenw10 if you need a tcpdump via the monitor port also.

Thank you

stephenw10

Did you import the config or configure it from scatch?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Did you import the config or configure it from scatch?

Imported the config during install phase, pfSense only asked to reconfigure the VLANs, the LAN and WAN interface, due to Interface mismatch.

Also I updated from 2.6.0->22.05. There is no update to 22.01 no more. I am afraid now to update the other box, due to the fact I cannot revert to 22.01.

Do you need any more tests from the new machine? I would like to revert it to it's initial use.

Do you think that if I reconfigure the interfaces on the previous machine will solve anything? This was the only thing I did on the new box, when asked by pfSense setup.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Did you import the config or configure it from scatch?

Let me show you another thing.
I will explain first. Every time I restarted pfSense on the new box the previous VLANs remained also, even if I specified the new ones. This triggered interface mismatch on each restart.

On ifconfig I found this:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0 prefixlen 64 scopeid 0x1
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        groups: DNS_Networks
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:54
        inet6 fe80::82ee:73ff:febb:e54%em0 prefixlen 64 scopeid 0x2
        inet ************ netmask 0xfffff800 broadcast *****************
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        inet 10.10.10.21 netmask 0xffffffff
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ix2.20: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
        ether 00:00:00:00:00:00
        inet6 fe80::82ee:73ff:febb:e55%ix2.20 prefixlen 64 tentative scopeid 0x7
        groups: vlan
        vlan: 0 vlanpcp: 0 parent interface: <none>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2.30: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
        ether 00:00:00:00:00:00
        inet6 fe80::82ee:73ff:febb:e55%ix2.30 prefixlen 64 tentative scopeid 0x8
        groups: vlan
        vlan: 0 vlanpcp: 0 parent interface: <none>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan DNS_Networks
        vlan: 20 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan DNS_Networks
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::82ee:73ff:febb:e55%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 59956

Look what vlan number is assigned to ix2.20, and ix2.30 (2 unexisting vlans).

After I deleted them, from the GUI->>Interfaces/VLANs there was no interface mismatch anymore.

stephenw10

Hmm, well either the igb driver is still stripping the tags or the ix driver is somehow adding them, but only for you...
We know that the igb/em driver doesn't pass vlan0 correctly because the ISPs that require it fail in 22.01 and 22.05. But it drops the packets rather than strips the tags.

Are you able to upload your config or status file from the Supermicro for us to review?

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, well either the igb driver is still stripping the tags or the ix driver is somehow adding them, but only for you...
We know that the igb/em driver doesn't pass vlan0 correctly because the ISPs that require it fail in 22.01 and 22.05. But it drops the packets rather than strips the tags.

Are you able to upload your config or status file from the Supermicro for us to review?

Steve

What is the "Status file" ?

I can provide the config to you guys, but only somewhere private. Can I exclude the passwords for OPENVPN and such by editing? Or it will be useless ? I will do whatever you ask, if you think we can solve this :)

stephenw10

The status_output file is a collection of redacted status and config data we use in support. You can get it by directly visiting the page: https://<your firewall IP>/status.php There is no link to it in the GUI.
The config included in that cannot be restored and tested as it's too heavily redacted so if you can include a separate config file, with whatever you need to remove removed, that will be very helpful.

You can upload files here: https://nc.netgate.com/nextcloud/s/QRNR6pPR2AmX6sN

Steve

NRgia

@stephenw10
Uploaded the files.
They're named:

"nrgia_super_micro_status_output.tar.gz"
"nrgia_supermicro_config-today-20220708193221.xml" - the config file

If something is missing due to editing, please let me know, and I can upload it again.

I hope all it's good in the config. I will wait for your conclusions.
Thank you

NRgia

@stephenw10
Sorry to bother you, did you had time to look over those files? Or, more time is needed? I can wait, no worries.
Thank you

stephenw10

Sorry I haven't had time yet. I should be able to review that shortly....

NRgia

@stephenw10
take your time
Thank you

NRgia

@stephenw10
Investigating the config file was indeed a good idea, and, in the end I found the culprit.
It's not pfSense, it's the Driver

In my config, I have done some NIC optimization a while ago, and for that I used shellcmd, to run it after boot.

You can see it here:

<shellcmd>ifconfig ix2 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum</shellcmd>

Now, to explain why I did the above.
In 2.6.0 or earlier, I don't remember exactly,(at least for my driver) Suricata did not pass traffic for VLANs, enabled on the parent interface. In order to solve that I wrote the above.

In 22.05 though, I had to add "vlanhwtag" again, otherwise even with Suricata uninstalled, no VLANs traffic will pass through.

I played with others tags, the only one that I must leave enabled is "vlanhwtag".

So the issues with the VLANs are solved, but now I cannot run Suricata in Inline mode with "vlanhwtag" enabled,(it will not pass traffic) or at least that is how it worked for me in previous versions.

Many thanks to @johnpoz and @stephenw10 for investigating with me.

stephenw10

Huh, that's fascinating. So as long as you do not disable hardware VLAN tagging on ix2 the VLANs work? And presumably a pcap no longer shows VLAN0 tagged traffic?

That's almost the opposite of what I'd expect....

I was never able to replicate it here.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Huh, that's fascinating. So as long as you do not disable hardware VLAN tagging on ix2 the VLANs work? And presumably a pcap no longer shows VLAN0 tagged traffic?

That's almost the opposite of what I'd expect....

I was never able to replicate it here.

The results are like this:

Disable "vlanhwtag" - the VLAN traffic will cease almost immediately, and also the pcap shows VLAN0 again.
Enable "vlanhwtag" - the VLAN traffic will work, and pcap will not show any VLAN0 anymore.

In 22.01 this did not happen with or without "vlanhwtag" .

The only inconvenient now is that I cannot run Suricata in Inline mode. I mean, I can, but the traffic for VLANS will not work. Only in legacy mode.

Other than that no complaints.

stephenw10

Wow. Boom!

20:30:18.349238 00:51:82:11:22:02 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 1001, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.101.0.1 tell 10.101.0.12, length 46

That's crazy! Time to dig deeper...

johnpoz

@stephenw10 so you were able to duplicate it then?

stephenw10

Yup. Bizarre. I would have sworn it wasn't real but...

johnpoz

@stephenw10 well guess you got a new "toy" to play with then - figuring out.. Can't wait to hear what you discover on what/how/why this happens..

So does this happen on other then ix nics? Could I remove the vlanhw setting on say an interface on my sg4860 and see this happening with vlan0? I assume you have already tried that?

stephenw10

Not yet, feel free
The same patch or something close to it went into all the Intel drivers so...possibly. Though it seems unlikely no-one would have hit it in igb.

Edit: igb in current 2.7 snapshots appears unaffected.

https://redmine.pfsense.org/issues/13381

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Not yet, feel free
The same patch or something close to it went into all the Intel drivers so...possibly. Though it seems unlikely no-one would have hit it in igb.

Edit: igb in current 2.7 snapshots appears unaffected.

https://redmine.pfsense.org/issues/13381

Wow, I'm glad that this ended up with a defect. It took a while :) Do you think this is something that can be fixed by Netgate or we should wait for an upstream fix?