pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@stephenw10

Last test as you instructed:

Added port 7 to VLAN 20 group and marked the port Untagged
Set PVID 20 to port 7
Started tcpdump on pfsense for ix2
Connected the cable to a desktop and to port 7

The desktop MAC is b4:2e:99:c7:b4:26

tcpdump.txt - sorry was too big

No lease, no internet

Maybe I should've removed port 7 from VLAN1 group also?
https://imgur.com/c5wOVI1

stephenw10

Hmm, well that's weird!

As long as you changed the PVID on port 7 to 20 then it would only be one way traffic anyway. I shouldn't prevent traffic VLAN20 working.

Were you able to take a pcap on the connected desktop?

It's hard to imagine what could be tagging that. It seems very unlikely the AP and switch would be doing so independently.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, well that's weird!

As long as you changed the PVID on port 7 to 20 then it would only be one way traffic anyway. I shouldn't prevent traffic VLAN20 working.

Were you able to take a pcap on the connected desktop?

It's hard to imagine what could be tagging that. It seems very unlikely the AP and switch would be doing so independently.

Steve

I thought you're off for today :)

I started to revert back to 22.01, in order to get things running again.

If you need pictures with the switch I can provide. I don't have any reason not to follow your suggestions. I'm the first who wants to solve this.

I tried to dump the desktop, but the interface was not getting any ip just a local one with 169...something...so I provided the ix2 one, dumped from pfsense.

I can reinstall 22.05 tomorrow if you are still willing, and still have ideas :)

stephenw10

Hmm, do you have Snort or Suricata in in-line mode on ix2 by any chance?

In 22.05 you have netmap enabled on ix2 (and ix3) and not in 22.01.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, do you have Snort or Suricata in in-line mode on ix2 by any chance?

In 22.05 you have netmap enabled on ix2 (and ix3) and not in 22.01.

The configuration is the same on both.
On 22.01 I have Suricata running in Inline mode yes, on ix2 and ix3.
On 22.05 Suricata it is installed but disabled on all interfaces for testing purposes.
Also pfblockerNG installed but disabled.
Besides Avahi and NUT I don't have anything else enabled as packages.

stephenw10

Hmm, did you reboot since disabling it? Is ix2 still showing netmap enabled?
There are known issues with netmap and VLANs and that's definitely a difference between your setup and mine. I'm trying to replicate it now...

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, did you reboot since disabling it? Is ix2 still showing netmap enabled?
There are known issues with netmap and VLANs and that's definitely a difference between your setup and mine. I'm trying to replicate it now...

I did reboot after I set PROMISC tag to see if it works. So at least one reboot.

stephenw10

Do you still see 'netmap' listed as an option on ix2 though?
Try setting the interface to legacy mode.

I can't replicate exactly what you're seeing but with Snort in-line enabled it cannot pass vlan traffic.

Just disabling the interface in Snort removes the netmap setting and allows vlan tagged traffic to pass here.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Do you still see 'netmap' listed as an option on ix2 though?
Try setting the interface to legacy mode.

I can't replicate exactly what you're seeing but with Snort in-line enabled it cannot pass vlan traffic.

Just disabling the interface in Snort removes the netmap setting and allows vlan tagged traffic to pass here.

Steve

After I disabled Suricata I did not saw Netmap on the interfaces. I paid close attention to tags, I was looking for "PROMISC". :)
But to exclude Suricata I can uninstall it altogether.
For now I reverted back to 22.01, but I can upgrade again tomorrow, and uninstall Suricata after the upgrade to 22.05. That is the easy part.

If you have other ideas you can post them, and I will try them in order and report back.

First I will try to set the interfaces to legacy, second I will uninstall the package.

stephenw10

That should be a good test. When I enable netmap here I see no VLAN tagged traffic arrive at all.

Do you have anything else running on the patent interface that might be setting it in promisc mode?

Steve

stephenw10

It would be good to repeat the pcap on ix2 with the VLAN working in 22.01. That will show us if the traffic is still tagged VLAN0 in your setup but the driver in 22.01 just allows it.

Steve

NRgia

This post is deleted!

NRgia

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@stephenw10
Besides Suricata, I do not run anything that should enable promisc mode. I don't think pfblocker enables, promisc mode.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

It would be good to repeat the pcap on ix2 with the VLAN working in 22.01. That will show us if the traffic is still tagged VLAN0 in your setup but the driver in 22.01 just allows it.

Steve

Run a test on pfSense 22.01 as instructed.

started tcpdump on ix2
Connected a mobile device to VLAN 20 with IP 192.168.10.57 with MAC 08:c5:e1:97:fa:ab

tcpdump_pfsense_22.01.txt

Any other tests on pfSense 22.01, or should I upgrade to 22.05 and continue with Suricata testing, like you proposed yesterday?

Thank you

stephenw10

Hmm, no vlan0 traffic at all there...
Does that mean the earlier driver is filtering it before the pcap can see it?
Or that the later driver is somehow adding it?

Are you able to create a mirror port on the switch so we can see what's actually on the wire?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, no vlan0 traffic at all there...
Does that mean the earlier driver is filtering it before the pcap can see it?

I don't know what to respond to this. I will let you draw any conclusions.

Or that the later driver is somehow adding it?

Are you able to create a mirror port on the switch so we can see what's actually on the wire?

I'm still on 22.01, I changed from Inline to Legacy Mode, and then disabled Suricata.
The NETMAP tag is not there anymore.

The test with the mirror port you want it done on 22.01 or 22.05?
And which port do you want me to mirror the pfSense LAN side?
And then what information do you want me to record, and how?
For example I will connect a device to that mirror port, and then? A tcpdump from pfsense, from the device, which interface?

Also do you want a full pcap file taken with wireshark, or with tcpdump like before?

stephenw10

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

The test with the mirror port you want it done on 22.01 or 22.05?

Both. Once it's configured you can use it to see what's on the wire in both situations.

And which port do you want me to mirror the pfSense LAN side?

Yes, the port linked to the pfSense LAN would be most useful there I think.

And then what information do you want me to record, and how?

Connect a client to the mirror port and run a pcap on that client. You will see everything that the pfSense LAN sees.

That will confirm which of those two suppositions is correct.
If you do see vlan0 tagged traffic there the we will know the older driver in 22.01 is actually stripping those tags allowing it to work.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

This is the the with Suricata enabled in inline mode:

tcp_dump_pfsense_22.01_Suricata_Inline_mode.txt

Same device 192.168.10.57 with MAC 08:c5:e1:97:fa:ab but there is other traffic also.

Next test, the mirror port. I will report shortly.

For the mirror port test, do you want me to leave Suricata enabled and the interface in legacy mode? Or disabled, to exclude it from interfering?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

Yes, the port linked to the pfSense LAN would be most useful there I think.

Test parameters:

pfSense 22.01
Suricata enabled on ix2 and ix3, in inline mode
Connected to mirror port which mirrors pfSense LAN side port
Did a tcpdump on connected device wired to the mirrored port.

This is the result:

tcpdump_port_mirroring_pfsense_22.01.txt

I cannot see any VLANs tags. I can only see them if I do a tcpdump from pfSense. Does this mean the switch strips the tags, and then add the tags only to tagged ports? (Which is correct behavior?)

Now I will upgrade to 22.05 if there are no tests to run on 22.01.

Please let me know if on 22.05 you want me to let Suricata run or should I set the interfaces to legacy and disable/uninstall Suricata?

NRgia

@stephenw10
Now for pfSense 22.05
First I set the interfaces to Legacy mode, disabled Suricata and uninstalled it.
Rebooted pfSense machine.
ifconfig shows like this after the reboot:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:88
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:89
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2 prefixlen 64 scopeid 0x3
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:8b
        inet6 fe80::ae1f:6bff:fe45:fa8b%ix3 prefixlen 64 scopeid 0x4
        inet *********** netmask 0xfffff800 broadcast 92.83.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ix2.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::ae1f:6bff:fe45:fa88%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 49792

My packages list

This is the tcpdump from pfSense machine on ix2 interface with suricata uninstalled:
tcpdump_from_pfsense_22.05_ix2.txt

Setting the interface to legacy and uninstalling Suricata did not help