VLAN Problem VIA based Nexgate NSA-1041 since 1.2.3 RC2

mibo

So here some informations

if you configure a VLAN it fells like no network connection. Interface (physical and vlan) is up, but no traffic.

i have seen that between RC1 and RC3 freebsd was updated to 7.2. and now i have more options in the interfaces than before. can a new default interface option cause this?

here my testing plan:

booting from LiveCD. No harddisk. working with defaults.
setup vlan0 on fxp1
setup lan on fxp0. connected to my notebook
setup wan on vlan0. connected to DHCP Server (company LAN)

now you can not reach any host behind WAN(vlan). it doesnt matter if i test on my notebook or in the ssh shell.

with freebsd 7.1 (1.2.3 RC1) it works.

infos from 1.2.3 release (not working):

uname -a
FreeBSD pfSense.local 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0: Sun Dec  6 23:20:31 EST 2009     
sullrich@FreeBSD_7.2_pfSense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7  i386

ifconfig
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19cf%fxp1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d0
        media: Ethernet autoselect (none)
        status: no carrier
fxp3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d1
        media: Ethernet autoselect (none)
        status: no carrier
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
        pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33204
vlan0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=3 <rxcsum,txcsum>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19ce%vlan0 prefixlen 64 scopeid 0x9
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 10 parent interface: fxp1</full-duplex></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></up,running></up,loopback,running,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

infos from 1.2.3 RC1 (working):

uname -a
FreeBSD pfSense.local 7.1-RELEASE-p5 FreeBSD 7.1-RELEASE-p5 #0: Wed Apr 22 15:35:36 EDT 2009     
sullrich@RELENG_1_2-snapshots.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7  i386

 ifconfig
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=b <rxcsum,txcsum,vlan_mtu>ether 00:10:f3:0e:19:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=b <rxcsum,txcsum,vlan_mtu>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19cf%fxp1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=b <rxcsum,txcsum,vlan_mtu>ether 00:10:f3:0e:19:d0
        media: Ethernet autoselect (none)
        status: no carrier
fxp3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=b <rxcsum,txcsum,vlan_mtu>ether 00:10:f3:0e:19:d1
        media: Ethernet autoselect (none)
        status: no carrier
plip0: flags=108810 <pointopoint,simplex,multicast,needsgiant>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33204
pfsync0: flags=41 <up,running>metric 0 mtu 1460
        pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
vlan0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19ce%vlan0 prefixlen 64 scopeid 0xa
        inet 192.168.3.110 netmask 0xfffffe00 broadcast 192.168.3.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 10 parent interface: fxp1</full-duplex></up,broadcast,running,simplex,multicast></up,running></promisc></up,loopback,running,multicast></pointopoint,simplex,multicast,needsgiant></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast>

Perry

@http://blog.pfsense.org/?p=531:

Warning for those using Intel PRO/100 cards – there is a regression in the fxp driver in FreeBSD 7.2 that may require disabling hardware checksum offloading under System -> Advanced if you have connectivity problems.

mibo

i have tried disabling hardware checksum offloading but no luck…

on fxp0 the RXSUM and TXSUM option disappeared, but on vlan0 and fxp1 not.

see ifconfig

fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19cf%fxp1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d0
        media: Ethernet autoselect (none)
        status: no carrier
fxp3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d1
        media: Ethernet autoselect (none)
        status: no carrier
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
        pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33204
vlan0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=3 <rxcsum,txcsum>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19ce%vlan0 prefixlen 64 scopeid 0x9
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 10 parent interface: fxp1</full-duplex></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></up,running></up,loopback,running,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

mibo

Switching the interfaces will switch the RXCSUM Options.

see ifconfig

fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:cf
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19cf%fxp1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d0
        media: Ethernet autoselect (none)
        status: no carrier
fxp3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:d1
        media: Ethernet autoselect (none)
        status: no carrier
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
        pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33204
vlan0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=3 <rxcsum,txcsum>ether 00:10:f3:0e:19:ce
        inet6 fe80::210:f3ff:fe0e:19ce%vlan0 prefixlen 64 scopeid 0x9
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 10 parent interface: fxp0</full-duplex></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></up,running></up,loopback,running,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></broadcast,simplex,multicast></full-duplex></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

Perry

You could try the manual way
ifconfig fxp0 -rxcsum -txcsum

mibo

sorry.. did not work. the option disappears but it still not work..

i had a look at a packet capture. At the DHCP discover on the WAN interface i can not see a VLAN tag.
other packet from my network come with a correct VLAN tag.

the vlan id is 10 and the following value appears in the ip packet:

Differentiated Services Field: 0x10 (DSCP 0x04: Unknown DSCP; ECN: 0x00)
0001 00.. = Differentiated Services Codepoint: Unknown (0x04)
.... ..0\. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0

i think the vlan tagging is not working correctly…

jimp

If it's not working properly, it must be something specific to your fxp chip revision.

I'm using VLANs heavily on fxp cards on 1.2.3-RELEASE and it works just fine for me, either with onboard NICs or add-in cards. It does look like yours have tso, you might try disabling that if it isn't already:

sysctl -w net.inet.tcp.tso=0

The output of these commands might also help:

# pciconf -lv | grep fxp0
fxp0@pci0:0:11:0:       class=0x020000 card=0x000c8086 chip=0x12298086 rev=0x08 hdr=0x00

# ifconfig -m fxp0
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2009 <rxcsum,vlan_mtu,wol_magic>capabilities=2049 <rxcsum,vlan_mtu,polling,wol_magic>ether 00:02:b3:03:80:28
        inet6 fe80::202:b3ff:fe03:8028%fxp0 prefixlen 64 scopeid 0x1 
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
                media none
                media 100baseTX mediaopt hw-loopback</full-duplex></rxcsum,vlan_mtu,polling,wol_magic></rxcsum,vlan_mtu,wol_magic></up,broadcast,running,simplex,multicast>

mibo

TSO is already disabled.

# sysctl -w net.inet.tcp.tso
net.inet.tcp.tso: 0

here the infos:

# pciconf -lv | grep fxp0
fxp0@pci0:0:9:0:        class=0x020000 card=0xffffffff chip=0x12098086 rev=0x10 hdr=0x00

# ifconfig -m fxp0
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>capabilities=21db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
                media none
                media 100baseTX mediaopt hw-loopback</full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_magic></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

jimp

Looks like yours has a later chip revision and claims to do VLANs in hardware. Mine doesn't claim such support, not sure any of them do.

Perhaps it's a problem along those lines, but unfortunately it's a bit beyond my level of expertise. It might be interesting to try a stock FreeBSD 7.2 install on there to see if you have similar problems, or it may be easier to try a pfSense 2.0 beta snapshot to see if the problem persists on FreeBSD 8.

mibo

So i have tested a pfSense 2.0 snapshot.
The same tests as before were made but the problem also exists in this version with the same behaviour.

to complete the tests here the outputs:

# ifconfig -m fxp0
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>capabilities=21db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
                media none
                media 100baseTX mediaopt hw-loopback</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_magic></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

# ifconfig
fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::210:f3ff:fe0e:19ce%fxp0 prefixlen 64 scopeid 0x1
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=2198 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic>ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19cf%fxp1 prefixlen 64 scopeid 0x2
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
...

fxp1_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 00:10:f3:0e:19:cf
        inet6 fe80::210:f3ff:fe0e:19ce%fxp1_vlan10 prefixlen 64 scopeid 0xa
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 10 parent interface: fxp1</full-duplex></performnud,accept_rtadv></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic></up,broadcast,running,simplex,multicast>

Any idea how to proceed?

I have found the following lines in the FreeBSD Release Notes:

The fxp(4) driver has been improved. Changes include:

Rx checksum offload support for 82559 or later controllers has been added

VLAN hardware tag insertion/stripping support and Tx/Rx checksum offload for VLAN frames support have been added. Note that the VLAN hardware assistance is available only on 82550 or 82551-based controllers.

pfSense recognize my card as a "Intel 82559ER". Can the VLAN hardware support be the issue? Like using VLAN hardware tag but my card cannot do this?

jimp

Try this:

ifconfig fxp0 -vlanhwtag -vlanhwfilter

Not sure about the hwfilter but try it anyhow, if it doesn't work, try just -vlanhwtag

That should disable the hardware tagging. If that works, install the shellcmd package and add a command in there for that. Might help for now.

mibo

Solved!
at first sight.

only disabling hwtag.

ifconfig fxp0 -vlanhwtag

I will do some more tests on the system till it goes back as my carp slave (with many VLANs ;-)).

Thanks a lot for your help!

Fakster

Hey mibo

would you know where I can find a pdf manual for the nsa 1041 ?

I have been given one but no instructions for hardware i.e. the pin header info
for power reset etc

any help appreciated, thanks

mibo

Hi Fakster,

have a look at the user manual:
http://www.nexcom.com/ProductModel_Download.aspx?id=97c0aec4-d471-49af-b4f5-78f12f0c57a8