pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, well either the igb driver is still stripping the tags or the ix driver is somehow adding them, but only for you...
We know that the igb/em driver doesn't pass vlan0 correctly because the ISPs that require it fail in 22.01 and 22.05. But it drops the packets rather than strips the tags.

Are you able to upload your config or status file from the Supermicro for us to review?

Steve

What is the "Status file" ?

I can provide the config to you guys, but only somewhere private. Can I exclude the passwords for OPENVPN and such by editing? Or it will be useless ? I will do whatever you ask, if you think we can solve this :)

stephenw10

The status_output file is a collection of redacted status and config data we use in support. You can get it by directly visiting the page: https://<your firewall IP>/status.php There is no link to it in the GUI.
The config included in that cannot be restored and tested as it's too heavily redacted so if you can include a separate config file, with whatever you need to remove removed, that will be very helpful.

You can upload files here: https://nc.netgate.com/nextcloud/s/QRNR6pPR2AmX6sN

Steve

NRgia

@stephenw10
Uploaded the files.
They're named:

"nrgia_super_micro_status_output.tar.gz"
"nrgia_supermicro_config-today-20220708193221.xml" - the config file

If something is missing due to editing, please let me know, and I can upload it again.

I hope all it's good in the config. I will wait for your conclusions.
Thank you

NRgia

@stephenw10
Sorry to bother you, did you had time to look over those files? Or, more time is needed? I can wait, no worries.
Thank you

stephenw10

Sorry I haven't had time yet. I should be able to review that shortly....

NRgia

@stephenw10
take your time
Thank you

NRgia

@stephenw10
Investigating the config file was indeed a good idea, and, in the end I found the culprit.
It's not pfSense, it's the Driver

In my config, I have done some NIC optimization a while ago, and for that I used shellcmd, to run it after boot.

You can see it here:

<shellcmd>ifconfig ix2 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum</shellcmd>

Now, to explain why I did the above.
In 2.6.0 or earlier, I don't remember exactly,(at least for my driver) Suricata did not pass traffic for VLANs, enabled on the parent interface. In order to solve that I wrote the above.

In 22.05 though, I had to add "vlanhwtag" again, otherwise even with Suricata uninstalled, no VLANs traffic will pass through.

I played with others tags, the only one that I must leave enabled is "vlanhwtag".

So the issues with the VLANs are solved, but now I cannot run Suricata in Inline mode with "vlanhwtag" enabled,(it will not pass traffic) or at least that is how it worked for me in previous versions.

Many thanks to @johnpoz and @stephenw10 for investigating with me.

stephenw10

Huh, that's fascinating. So as long as you do not disable hardware VLAN tagging on ix2 the VLANs work? And presumably a pcap no longer shows VLAN0 tagged traffic?

That's almost the opposite of what I'd expect....

I was never able to replicate it here.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Huh, that's fascinating. So as long as you do not disable hardware VLAN tagging on ix2 the VLANs work? And presumably a pcap no longer shows VLAN0 tagged traffic?

That's almost the opposite of what I'd expect....

I was never able to replicate it here.

The results are like this:

Disable "vlanhwtag" - the VLAN traffic will cease almost immediately, and also the pcap shows VLAN0 again.
Enable "vlanhwtag" - the VLAN traffic will work, and pcap will not show any VLAN0 anymore.

In 22.01 this did not happen with or without "vlanhwtag" .

The only inconvenient now is that I cannot run Suricata in Inline mode. I mean, I can, but the traffic for VLANS will not work. Only in legacy mode.

Other than that no complaints.

stephenw10

Wow. Boom!

20:30:18.349238 00:51:82:11:22:02 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 1001, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.101.0.1 tell 10.101.0.12, length 46

That's crazy! Time to dig deeper...

johnpoz

@stephenw10 so you were able to duplicate it then?

stephenw10

Yup. Bizarre. I would have sworn it wasn't real but...

johnpoz

@stephenw10 well guess you got a new "toy" to play with then - figuring out.. Can't wait to hear what you discover on what/how/why this happens..

So does this happen on other then ix nics? Could I remove the vlanhw setting on say an interface on my sg4860 and see this happening with vlan0? I assume you have already tried that?

stephenw10

Not yet, feel free
The same patch or something close to it went into all the Intel drivers so...possibly. Though it seems unlikely no-one would have hit it in igb.

Edit: igb in current 2.7 snapshots appears unaffected.

https://redmine.pfsense.org/issues/13381

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Not yet, feel free
The same patch or something close to it went into all the Intel drivers so...possibly. Though it seems unlikely no-one would have hit it in igb.

Edit: igb in current 2.7 snapshots appears unaffected.

https://redmine.pfsense.org/issues/13381

Wow, I'm glad that this ended up with a defect. It took a while :) Do you think this is something that can be fixed by Netgate or we should wait for an upstream fix?

stephenw10

Almost certainly upstream. Need to confirm it still exists in FreeBSD head first though.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Almost certainly upstream. Need to confirm it still exists in FreeBSD head first though.

Thanks again for the support, let's hope it will not take 5 years :) . Yes I know, it's out of your hands.

stephenw10

Mmm, this is the most interesting issue I've seen in a while!

You need to disable vlan hardware tagging for Suricata to be able to see the traffic? Is that is you run Suricata on the parent NIC only?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Mmm, this is the most interesting issue I've seen in a while!

You need to disable vlan hardware tagging for Suricata to be able to see the traffic? Is that is you run Suricata on the parent NIC only?

Exactly, I'll try to explain better.
For 22.01 I found a workaround in order for the Suricata package to not break VLAN traffic.

For that I disabled the vlanhwtag.

If I left vlanhwtag enabled, only traffic from the parrent interface will pass.

Suricata is enabled only on the parent interface, as @bmeeks recommends.

On 22.05 I cannot apply the same workaround anymore, due to, when I disable vlanhwtag, that VLAN0 will appear and VLANs traffic will cease, even with Suricata uninstalled.

NRgia

@stephenw10
I saw you commented on this defect
https://redmine.pfsense.org/issues/13381

that you tested on FreeBSD 14

The behavior is the same?