SG-1000 VLANs not working unless PROMISC is set

Rocco83

Hi Steve,

I have set PROMISC on cpsw0 and cpsw1.

The LAGG is LACP configured.
Without PROMISC on cpsw*, one port of the LACP (cpsw0 OR cpsw1) was never distributing.
It was one at the time, normally cpsw1, but if i take out of the LAGG cpsw0 then cpsw1 is working.

Can you test traffic across the LAGG directly without VLANs?

This is working, it's affecting only the VLANs.

Thanks,
Daniele

stephenw10

Yes, I don't think that's a supported mode.
The VLANs will apply settings to their parent interface but if that's a LAGG that is normally passed to it's members. But the cpsw driver is somewhat unique and may not work with that.
If you need to use the setup I suspect you will have to add the ifconfig command as a shellcmd to correct it.
https://docs.netgate.com/pfsense/en/latest/development/executing-commands-at-boot-time.html

If you use the afterfilterchnagesshellcmd type it should reapply when you make changes so not come out of promiscuous mode.

Steve

Rocco83

About the supported mode,
I was effectively using it.
Then i have upgraded && possibly changed something, and pfsense got broken.
So it was at least working :)

I have made a further test, and this really seems not linked to the LAGG configuration.
I have removed cpsw1 from the lagg, and moved to another switch physical port.
Added the VLAN on top of it, and assigned to a network interface in pfsense.

[root@pf2-tos ~]# ping 172.16.82.241
PING 172.16.82.241 (172.16.82.241): 56 data bytes
^C
--- 172.16.82.241 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[root@pf2-tos ~]# ifconfig cpsw1 promisc
[root@pf2-tos ~]# ping 172.16.82.241
PING 172.16.82.241 (172.16.82.241): 56 data bytes
64 bytes from 172.16.82.241: icmp_seq=0 ttl=64 time=1.447 ms
64 bytes from 172.16.82.241: icmp_seq=1 ttl=64 time=0.697 ms
^C
--- 172.16.82.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.697/1.072/1.447/0.375 ms
[root@pf2-tos ~]#

[root@pf2-tos ~]# ifconfig cpsw1
cpsw1: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
ether c8:df:84:c1:16:39
hwaddr c8:df:84:c1:16:39
inet6 fe80::cadf:84ff:fec1:1639%cpsw1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[root@pf2-tos ~]# ifconfig cpsw1.12
cpsw1.12: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether c8:df:84:c1:16:39
inet6 fe80::cadf:84ff:fec1:1639%cpsw1.12 prefixlen 64 scopeid 0x10
inet 172.16.82.242 netmask 0xffffff00 broadcast 172.16.82.255
groups: vlan
vlan: 12 vlanpcp: 0 parent interface: cpsw1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[root@pf2-tos ~]#

So seems not bonded to the bonding configuration (ah, i like stupid wording jokes :D)

stephenw10

Ah, interesting. Do you know what version you were running when it was working as expected?

Steve

Rocco83

I recall that the first issue was back in Apr/2019.
According to https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html,
I was moving from 2.4.4-p1 to 2.4.4-p2.
I'm now on 2.4.4-p3.

But, I'm not 100% sure about it...

Back to https://redmine.pfsense.org/issues/7645, what was the changelog for the issue?
Just wondering to identify if there is a regression.

Thanks,
Daniele

stephenw10

Just spent a while looking for it and failed to track it down. I'll have to ask someone who might know directly.

Rocco83

Hi there,

Any news?

Thank you very much,
Daniele

stephenw10

I've been unable to replicate this in 2.4.4p3 or 2.5.

[2.4.4-RELEASE][root@ufw3.stevew.lan]/root: ifconfig
cpsw0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether 0c:b2:b7:af:2f:4f
        hwaddr 0c:b2:b7:af:2f:4f
        inet6 fe80::eb2:b7ff:feaf:2f4f%cpsw0 prefixlen 64 scopeid 0x1 
        inet 172.21.16.80 netmask 0xffffff00 broadcast 172.21.16.255 
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
cpsw1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether 0c:b2:b7:af:2f:51
        hwaddr 0c:b2:b7:af:2f:51
        inet 192.168.80.1 netmask 0xffffff00 broadcast 192.168.80.255 
        inet6 fe80::1:1%cpsw1 prefixlen 64 duplicated scopeid 0x2 
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 224.0.0.240 maxupd: 128 defer: on
        syncok: 1
        groups: pfsync 
pflog0: flags=100<PROMISC> metric 0 mtu 33184
        groups: pflog 
cpsw1.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 0c:b2:b7:af:2f:51
        inet6 fe80::eb2:b7ff:feaf:2f51%cpsw1.50 prefixlen 64 scopeid 0x7 
        inet 172.18.10.11 netmask 0xffffff00 broadcast 172.18.10.255 
        groups: vlan 
        vlan: 50 vlanpcp: 0 parent interface: cpsw1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[2.4.4-RELEASE][root@ufw3.stevew.lan]/root: ping 172.18.10.1
PING 172.18.10.1 (172.18.10.1): 56 data bytes
64 bytes from 172.18.10.1: icmp_seq=0 ttl=64 time=0.621 ms
64 bytes from 172.18.10.1: icmp_seq=1 ttl=64 time=0.972 ms
64 bytes from 172.18.10.1: icmp_seq=2 ttl=64 time=0.491 ms
^C
--- 172.18.10.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.491/0.695/0.972/0.203 ms

Capturing at the other side:

16:36:42.590615 0c:b2:b7:af:2f:51 > 00:90:0b:76:8e:52, ethertype 802.1Q (0x8100), length 60: vlan 50, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 9584, offset 0, flags [none], proto ICMP (1), length 28)
    172.18.10.11 > 172.18.10.1: ICMP echo request, id 52423, seq 628, length 8
16:36:42.590667 00:90:0b:76:8e:52 > 0c:b2:b7:af:2f:51, ethertype 802.1Q (0x8100), length 46: vlan 50, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 59789, offset 0, flags [none], proto ICMP (1), length 28)
    172.18.10.1 > 172.18.10.11: ICMP echo reply, id 52423, seq 628, length 8

Requiring promiscuous mode like that usually implies the wrong MAC address or at least an unexpected MAC. I wonder if the lagg on there altered it for that driver?

Steve

Rocco83

@stephenw10 lagg has been not configured. Without lagg I confirm that this is working

My lagg is configred with lacp

Thanks,

stephenw10

Oh, OK so this only a problem over lagg? In your previous post it looked like you removed cpsw1 from the lagg and a VLAN on that port still only worked with promiscuous mode enabled.

Steve

Rocco83

Shame on myself,

I have made a further test, and this really seems not linked to the LAGG configuration.
So, i'll try to upgrade it from scratch and let's see.

I will keep you posted.

Thanks,
Daniele

stephenw10

Ah, OK, so possibly an issue with the test outside of the lagg?

I'll see if I can replicate the lagg issue.

Steve

Rocco83

Fast forward to Q3 2021, finally i had some time to reproduce everything.

resetted the SG-1000, installed the pfsense plus 21.02.2
bringed up ssh
create a rule to allow everything in floating (with quick)
NOT configured the lagg
LAN on cpsw1
setup one vlan per NIC (cpsw0.10, cpsw1.11)

tests:

reboot
ping is working on LAN (untagged) but not not VLANs (tagged)
ifconfig promisc on cpsw0
ping is working on both cpsw0.10 AND cpsw1.11
reboot
ping is working on LAN (untagged) but not not VLANs (tagged)
ifconfig promisc on cpsw1
ping is working on both cpsw0.10 AND cpsw1.11

reproducer:

*** Welcome to Netgate pfSense Plus 21.02.2-RELEASE (arm) on pf2-tos ***

 WAN (wan)       -> cpsw1.11   -> v4: 172.16.81.242/24
 LAN (lan)       -> cpsw1      -> v4: 172.16.8.242/24
 IOT (opt1)      -> cpsw0.10   -> v4: 172.16.80.242/24

initial status after boot, ping not working on VLANs, but working on main LAN interface

[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.8.241
PING 172.16.8.241 (172.16.8.241): 56 data bytes
64 bytes from 172.16.8.241: icmp_seq=0 ttl=64 time=1.292 ms
64 bytes from 172.16.8.241: icmp_seq=1 ttl=64 time=0.738 ms

--- 172.16.8.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.738/1.015/1.292/0.277 ms 
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.80.241
PING 172.16.80.241 (172.16.80.241): 56 data bytes

--- 172.16.80.241 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.81.241
PING 172.16.81.241 (172.16.81.241): 56 data bytes

--- 172.16.81.241 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root:

Setting promisc on cpsw0, ping working on both interfaces

[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw0 promisc
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.80.241
PING 172.16.80.241 (172.16.80.241): 56 data bytes
64 bytes from 172.16.80.241: icmp_seq=0 ttl=64 time=1.501 ms
64 bytes from 172.16.80.241: icmp_seq=1 ttl=64 time=0.889 ms

--- 172.16.80.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.889/1.195/1.501/0.306 ms
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.81.241
PING 172.16.81.241 (172.16.81.241): 56 data bytes
64 bytes from 172.16.81.241: icmp_seq=0 ttl=64 time=1.062 ms
64 bytes from 172.16.81.241: icmp_seq=1 ttl=64 time=0.892 ms

--- 172.16.81.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.892/0.977/1.062/0.085 ms
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root:

reboot, to enable promisc on cpsw1

ping not working at boot

21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.8.241
PING 172.16.8.241 (172.16.8.241): 56 data bytes
64 bytes from 172.16.8.241: icmp_seq=0 ttl=64 time=1.309 ms
64 bytes from 172.16.8.241: icmp_seq=1 ttl=64 time=0.713 ms

--- 172.16.8.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.713/1.011/1.309/0.298 ms
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.80.241
PING 172.16.80.241 (172.16.80.241): 56 data bytes

--- 172.16.80.241 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.81.241
PING 172.16.81.241 (172.16.81.241): 56 data bytes

--- 172.16.81.241 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root:

ifconfig cpsw1 promisc, ping working

[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw1 promisc
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.80.241
PING 172.16.80.241 (172.16.80.241): 56 data bytes
64 bytes from 172.16.80.241: icmp_seq=0 ttl=64 time=1.429 ms
64 bytes from 172.16.80.241: icmp_seq=1 ttl=64 time=0.756 ms

--- 172.16.80.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.756/1.092/1.429/0.337 ms
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ping -c2 172.16.81.241
PING 172.16.81.241 (172.16.81.241): 56 data bytes
64 bytes from 172.16.81.241: icmp_seq=0 ttl=64 time=0.993 ms
64 bytes from 172.16.81.241: icmp_seq=1 ttl=64 time=0.823 ms

--- 172.16.81.241 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.823/0.908/0.993/0.085 ms
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root:

settings:

[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw0
cpsw0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether c8:df:84:c1:16:37
        inet6 fe80::cadf:84ff:fec1:1637%cpsw0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw1
cpsw1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether c8:df:84:c1:16:39
        inet6 fe80::cadf:84ff:fec1:1639%cpsw1 prefixlen 64 scopeid 0x2
        inet 172.16.8.242 netmask 0xffffff00 broadcast 172.16.8.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw1.11
cpsw1.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=80000<LINKSTATE>
        ether c8:df:84:c1:16:39
        inet6 fe80::cadf:84ff:fec1:1639%cpsw1.11 prefixlen 64 scopeid 0x7
        inet 172.16.81.242 netmask 0xffffff00 broadcast 172.16.81.255
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: cpsw1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root: ifconfig cpsw0.10
cpsw0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IOT
        options=80000<LINKSTATE>
        ether c8:df:84:c1:16:37
        inet6 fe80::cadf:84ff:fec1:1637%cpsw0.10 prefixlen 64 scopeid 0x8
        inet 172.16.80.242 netmask 0xffffff00 broadcast 172.16.80.255
        groups: vlan
        vlan: 10 vlanpcp: 0 parent interface: cpsw0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[21.02.2-RELEASE][root@pf2-tos.gonzaga.retaggio.net]/root:

This could be easily reproduced, in theory...

Rocco83

On the other side, the tcpdump show:

[2.4.5-RELEASE][admin@pf1-tos.domain]/root: tcpdump -i xn0.10 ether host c8:df:84:c1:16:37 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xn0.10, link-type EN10MB (Ethernet), capture size 262144 bytes
18:05:55.841165 ARP, Request who-has 172.16.80.241 tell 172.16.80.242, length 42
18:05:55.841190 ARP, Reply 172.16.80.241 is-at 00:16:3e:19:25:5f, length 28
18:05:56.850828 ARP, Request who-has 172.16.80.241 tell 172.16.80.242, length 42
18:05:56.850855 ARP, Reply 172.16.80.241 is-at 00:16:3e:19:25:5f, length 28

i can't run it on the origin otherwise the ping start working...