pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

Yeah it seems unlikely to be the driver because that's the same SoC with the same NICs we use in the 6100 and 7100. And the 7100 uses VLANs on it by default.

Kernel modules from 22.01 will not load in the 22.05 kernel. At least most won't, especially something like that. You would have to build the 22.01 driver against 22.05. And it probably won't build without some work. You might be able to just revert the patch we think went in the created the issue in your network and compile that. However if it does that only proves the VLAN0 handling was bad.

We need to see pcap on a mirror port showing all the traffic on the wire going into the port with working two way connectivity.

It almost impossible to believe the driver could add those tags to incoming packets. Incorrectly removing them in 22.01 is far more likely.

Steve

johnpoz

@stephenw10 I just don't see how the driver would add them as well - that just makes no sense. And if that was the case as you mention you have it on a bunch of devices sold by netgate. And hard to believe he is the only one running this specific MB etc. But the driver would be the same.. So why would add something in on his, but not all the others?

So what would be something different on his hardware where something wrong with drive adding the vlan 0, but nothing else is - I would think lots of people running 22.05 with ix interfaces - why is the board not a flame with people saying their vlans are not working.

This is something pretty unique in this setup that is causing it.. Just missing what that something is in trying to solve the puzzle.

btw: I will fire up the mirror port on my netgear tmrw, unless something breaks or catches fire I have a pretty empty cal tmrw with real life work ;)

stephenw10

Yup, I agree. I seems far more likely we are just failing to capture the offending traffic at this point.

Except it appeared to do the same thing with the AP connected directly to ix2.
So now I'm starting to doubt things!

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

it seems unlikely to be the driver because that's the same SoC with the same NICs we use in the 6100 and 7100. And the 7100 uses VLANs on it by default.

I don't know if you have the management LAN, on Native LAN, you have it also on VLAN, like VLAN 10 or something? Maybe you can try replicate my setup, although you don't have the same switch. Or maybe I ask to much, it was just an idea.

I mean is there any mistake in this picture for LAN:
https://imgur.com/a/WIZZ6rB
?

If I can do something else, let me know

stephenw10

That should work fine. You do have spare NICs so you could try other setups like putting VLANs 20 and 30 on ix1 and running a second link to the switch. But it should work as you have it now in 22.05.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

That should work fine. You do have spare NICs so you could try other setups like putting VLANs 20 and 30 on ix1 and running a second link to the switch. But it should work as you have it now in 22.05.

I think I can repurpose another mini pc that uses igb and em drivers(for a few hours). But I will need some time to save my work. Then just to understand what it will prove:

If it works on another machine with other type of NICs then it's not the switch.
If it doesn't it means it's the switch?
I'm trying to double check with you on this, to make sure this will not be in vain, because what I see here https://github.com/pfsense/FreeBSD-src/commit/9c762cc125c0c2dae9fbf49cc526bb97c14b54a4 is that the fix is also for igb and em drivers.

stephenw10

Yes if the switch is really tagging the traffic with VLAN0 it should fail using any NIC/driver in 22.05.

Though as I understand it we also saw that when the switch was removed entirely and the AP was connected directly to pfSense. Which is hard to explain.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Yes if the switch is really tagging the traffic with VLAN0 it should fail using any NIC/driver in 22.05.

Though as I understand it we also saw that when the switch was removed entirely and the AP was connected directly to pfSense. Which is hard to explain.

Yes, that's correct, that was one of the test you asked me to do. The AP was connected directly to pfSense. That's why I don't know what to say, if it's the switch or not. If I understood it corectly @johnpoz has almost the same switch as mine and also the same AP ( Flex HD) and he did not encounter any issues. I don't know if he had a board that used ix driver. Other then that he has 2 of my possible ofending devices.

johnpoz

@nrgia yeah I had no issues with vlans, and a native vlan as well. Both with port qos and 802.1p.

Only difference is yeah my switch is a bit different, baby sister to your switch. And mine is downstream of another switch that could be stripping the vlan 0??

But if you said you plugged the AP directly into the interface on pfsense, that kind of rules out the switch.

This is a odd one for sure.. tmrw I could plug my AP directly into a port on pfsense, its just on a ix interface its a igb interface.

stephenw10

Mmm, because we also saws the vlan0 tags when we removed the access point and just used the switch.
That implies it's either not the AP or the switch or that it's both devices doing it. Which seems unlikely!

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Mmm, because we also saws the vlan0 tags when we removed the access point and just used the switch.
That implies it's either not the AP or the switch or that it's both devices doing it. Which seems unlikely!

I got news. @stephenw10 and @johnpoz
I installed pfSense 22.05 on the another mini PC that has 2 NICS that uses igb and em drivers.

This is some info from ifconfig:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0 prefixlen 64 scopeid 0x1
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:54
        inet6 fe80::82ee:73ff:febb:e54%em0 prefixlen 64 scopeid 0x2
        inet ************* netmask 0xfffff800 broadcast ***************
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        inet 10.10.10.21 netmask 0xffffffff
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
igb0.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::82ee:73ff:febb:e55%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 59956

Now a tcpdump from pfSense igb0 - the new LAN interface with NATIVE, VLAN20 and VLAN30
pfsense_lan_tcpdump_new_machine.txt

Everything works, every WLAN, every VLAN, any host. No problem whatsoever.

Conclusions please guys

Tell me @stephenw10 if you need a tcpdump via the monitor port also.

Thank you

stephenw10

Did you import the config or configure it from scatch?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Did you import the config or configure it from scatch?

Imported the config during install phase, pfSense only asked to reconfigure the VLANs, the LAN and WAN interface, due to Interface mismatch.

Also I updated from 2.6.0->22.05. There is no update to 22.01 no more. I am afraid now to update the other box, due to the fact I cannot revert to 22.01.

Do you need any more tests from the new machine? I would like to revert it to it's initial use.

Do you think that if I reconfigure the interfaces on the previous machine will solve anything? This was the only thing I did on the new box, when asked by pfSense setup.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Did you import the config or configure it from scatch?

Let me show you another thing.
I will explain first. Every time I restarted pfSense on the new box the previous VLANs remained also, even if I specified the new ones. This triggered interface mismatch on each restart.

On ifconfig I found this:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0 prefixlen 64 scopeid 0x1
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        groups: DNS_Networks
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 80:ee:73:bb:0e:54
        inet6 fe80::82ee:73ff:febb:e54%em0 prefixlen 64 scopeid 0x2
        inet ************ netmask 0xfffff800 broadcast *****************
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        inet 10.10.10.21 netmask 0xffffffff
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ix2.20: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
        ether 00:00:00:00:00:00
        inet6 fe80::82ee:73ff:febb:e55%ix2.20 prefixlen 64 tentative scopeid 0x7
        groups: vlan
        vlan: 0 vlanpcp: 0 parent interface: <none>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2.30: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
        ether 00:00:00:00:00:00
        inet6 fe80::82ee:73ff:febb:e55%ix2.30 prefixlen 64 tentative scopeid 0x8
        groups: vlan
        vlan: 0 vlanpcp: 0 parent interface: <none>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan DNS_Networks
        vlan: 20 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether 80:ee:73:bb:0e:55
        inet6 fe80::82ee:73ff:febb:e55%igb0.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan DNS_Networks
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::82ee:73ff:febb:e55%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 59956

Look what vlan number is assigned to ix2.20, and ix2.30 (2 unexisting vlans).

After I deleted them, from the GUI->>Interfaces/VLANs there was no interface mismatch anymore.

stephenw10

Hmm, well either the igb driver is still stripping the tags or the ix driver is somehow adding them, but only for you...
We know that the igb/em driver doesn't pass vlan0 correctly because the ISPs that require it fail in 22.01 and 22.05. But it drops the packets rather than strips the tags.

Are you able to upload your config or status file from the Supermicro for us to review?

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, well either the igb driver is still stripping the tags or the ix driver is somehow adding them, but only for you...
We know that the igb/em driver doesn't pass vlan0 correctly because the ISPs that require it fail in 22.01 and 22.05. But it drops the packets rather than strips the tags.

Are you able to upload your config or status file from the Supermicro for us to review?

Steve

What is the "Status file" ?

I can provide the config to you guys, but only somewhere private. Can I exclude the passwords for OPENVPN and such by editing? Or it will be useless ? I will do whatever you ask, if you think we can solve this :)

stephenw10

The status_output file is a collection of redacted status and config data we use in support. You can get it by directly visiting the page: https://<your firewall IP>/status.php There is no link to it in the GUI.
The config included in that cannot be restored and tested as it's too heavily redacted so if you can include a separate config file, with whatever you need to remove removed, that will be very helpful.

You can upload files here: https://nc.netgate.com/nextcloud/s/QRNR6pPR2AmX6sN

Steve

NRgia

@stephenw10
Uploaded the files.
They're named:

"nrgia_super_micro_status_output.tar.gz"
"nrgia_supermicro_config-today-20220708193221.xml" - the config file

If something is missing due to editing, please let me know, and I can upload it again.

I hope all it's good in the config. I will wait for your conclusions.
Thank you

NRgia

@stephenw10
Sorry to bother you, did you had time to look over those files? Or, more time is needed? I can wait, no worries.
Thank you

stephenw10

Sorry I haven't had time yet. I should be able to review that shortly....