Speed Interface error in QinQ interface
-
Hi,
I have 2 Chelsio 4x10Gb SFP+ (T540-LP-CR) and I configured 3 laggs (failover). Then I configured vlan 100 on lagg2 and then I configured several QinQ interface on top vlan 100. Now everything works, but in the dashboard the speed of the QinQ interface is:
1000baseT <full-duplex>
instead of
10Gbase-LR <full-duplex>
The clx interfaces of the lagg2 (cxl2,cxl6 ) have the correct media:
10Gbase-LR <full-duplex>
this is my ifconfig output:
cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f0 hwaddr 00:07:43:56:71:f0 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f8 hwaddr 00:07:43:56:71:f8 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=8c04b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:72:00 hwaddr 00:07:43:56:72:00 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:56:72:08 hwaddr 00:07:43:56:72:08 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: no carrier cxl4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f0 hwaddr 00:07:43:56:72:90 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f8 hwaddr 00:07:43:56:72:98 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=8c04b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:72:00 hwaddr 00:07:43:56:72:a0 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: active cxl7: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:56:72:a8 hwaddr 00:07:43:56:72:a8 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> status: no carrier enc0: flags=0<> metric 0 mtu 1536 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: enc lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo pflog0: flags=100<PROMISC> metric 0 mtu 33160 lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f0 inet6 fe80::207:43ff:fe56:71f0%lagg0 prefixlen 64 scopeid 0xf inet a.b.c.d netmask 0xfffffff0 broadcast 46.44.246.159 inet a.b.c.e netmask 0xfffffff0 broadcast 46.44.246.159 vhid 1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active carp: MASTER vhid 1 advbase 1 advskew 0 groups: lagg laggproto failover lagghash l2,l3,l4 laggport: cxl0 flags=5<MASTER,ACTIVE> laggport: cxl4 flags=0<> lagg1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=8c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:71:f8 inet6 fe80::207:43ff:fe56:71f8%lagg1 prefixlen 64 scopeid 0x10 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto failover lagghash l2,l3,l4 laggport: cxl1 flags=5<MASTER,ACTIVE> laggport: cxl5 flags=0<> lagg2: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1516 options=8c04b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,VLAN_HWTSO,LINKSTATE> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2 prefixlen 64 scopeid 0x11 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto failover lagghash l2,l3,l4 laggport: cxl2 flags=5<MASTER,ACTIVE> laggport: cxl6 flags=0<> lagg1.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=80000<LINKSTATE> ether 00:07:43:56:71:f8 inet6 fe80::207:43ff:fe56:71f8%lagg1.6 prefixlen 64 scopeid 0x12 inet l.m.n.o netmask 0xffffffe0 broadcast 79.135.45.127 inet l.m.n.p netmask 0xffffffe0 broadcast 79.135.45.127 vhid 2 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 6 vlanpcp: 0 parent interface: lagg1 carp: MASTER vhid 2 advbase 1 advskew 0 groups: vlan lagg1.7: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=80000<LINKSTATE> ether 00:07:43:56:71:f8 inet6 fe80::207:43ff:fe56:71f8%lagg1.7 prefixlen 64 scopeid 0x13 inet q.r.s.t netmask 0xffffff00 broadcast 91.201.104.255 inet q.r.s.t.u netmask 0xffffff00 broadcast 91.201.104.255 vhid 3 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 7 vlanpcp: 0 parent interface: lagg1 carp: MASTER vhid 3 advbase 1 advskew 0 groups: vlan lagg2.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=80400<LRO,LINKSTATE> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100 prefixlen 64 scopeid 0x14 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 100 vlanpcp: 0 parent interface: lagg2 groups: vlan QinQ lagg2.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=80400<LRO,LINKSTATE> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100 prefixlen 64 scopeid 0x14 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 100 vlanpcp: 0 parent interface: lagg2 groups: vlan QinQ lagg2.100.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.20 prefixlen 64 scopeid 0x15 inet 10.0.0.1 netmask 0xffffffe0 broadcast 10.0.0.31 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.21: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.21 prefixlen 64 scopeid 0x16 inet 10.0.1.1 netmask 0xfffffff8 broadcast 10.0.1.7 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.22: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.22 prefixlen 64 scopeid 0x17 inet 10.0.1.65 netmask 0xfffffff8 broadcast 10.0.1.71 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.23: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.23 prefixlen 64 tentative scopeid 0x18 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.24: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.24 prefixlen 64 scopeid 0x19 inet 10.0.1.73 netmask 0xfffffff8 broadcast 10.0.1.79 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.25: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.25 prefixlen 64 scopeid 0x1a inet 10.0.1.81 netmask 0xfffffff8 broadcast 10.0.1.87 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.26: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.26 prefixlen 64 scopeid 0x1b inet 10.0.1.89 netmask 0xfffffff8 broadcast 10.0.1.95 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ lagg2.100.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1516 options=28<VLAN_MTU,JUMBO_MTU> ether 00:07:43:56:72:00 inet6 fe80::207:43ff:fe56:7200%lagg2.100.100 prefixlen 64 scopeid 0x1c inet 10.255.255.1 netmask 0xffffffe0 broadcast 10.255.255.31 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active groups: QinQ
the questions are:
-
Why the QinQ interfaces media is 1000BaseT instead 10000Base-SR?
-
Is it correct that lagg0 (WAN) lagg1 (other net) media is "autoselect" and not 10000Base-SR?
Thank you
-
-
@prx said in Speed Interface error in QinQ interface:
ifconfig
Hi,
Don't you usually disable LRO, TSO stuff?
These are not good if you are a router or only, if you are an endpoint...
https://calomel.org/freebsd_network_tuning.htmlalthough even automatic speed negotion should work, if the setting on the other side is "auto"
+++edit:
also important:
Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
This tunable must be set according to your configuration. VERY IMPORTANT!
Set FC to 0 (<x>) on all interfaces
hw.igb.<x>.fc=0 #Also put this in System Tunables hw.igb.<x>.fc: value=0
and / or
Disable Energy Efficiency - set for each igb port in your system
This setting can cause Link flap errors if not disabled
Set for every igb interface in the system as per these examples
dev.igb.0.eee_disabled: value=1
dev.igb.1.eee_disabled: value=1
dev.igb.2.eee_disabled: value=1
dev.igb.3.eee_disabled: value=1 -
Hi,
as you can see from the attachment, I disabled:
- Hardware Checksum Offloading
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
@DaddyGo
My NICs are 2 Chelsio T540-LP-CR (every nic has 4x10Gb SFP+ ports), so if I understood well, I have to add the following "lines" in Advanced->System Tunables:- hw.cxl0.fc value 0
- hw.cxl1.fc value 0
- hw.cxl2.fc value 0
- hw.cxl3.fc value 0
- hw.cxl4.fc value 0
- hw.cxl5.fc value 0
- hw.cxl6.fc value 0
- hw.cxl7.fc value 0
and
- dev.cxl.0.eee_disabled value 1
- dev.cxl.1.eee_disabled value 1
- dev.cxl.2.eee_disabled value 1
- dev.cxl.3.eee_disabled value 1
- dev.cxl.4.eee_disabled value 1
- dev.cxl.5.eee_disabled value 1
- dev.cxl.6.eee_disabled value 1
- dev.cxl.7.eee_disabled value 1
is it right?
The speed negotiation is set on "auto" on the switches (Cisco nexus 3064-X). This is the running conf of the interface:
interface Ethernet1/12 description Internal QinQ Vlans switchport mode trunk switchport trunk allowed vlan 99-120 spanning-tree port type edge trunk
And this is the output of show interface ethernet 1/12:
Ethernet1/12 is up admin state is up, Dedicated Interface Hardware: 100/1000/10000 Ethernet, address: 0006.f618.91ee (bia 0006.f618.91ee) Description: Internal QinQ Vlans MTU 1500 bytes, BW 10000000 Kbit , DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, medium is broadcast Port mode is trunk full-duplex, 10 Gb/s, media type is 10G Beacon is turned off Auto-Negotiation is turned on FEC mode is Auto Input flow-control is off, output flow-control is off Auto-mdix is turned off Rate mode is dedicated Switchport monitor is off EtherType is 0x8100 EEE (efficient-ethernet) : n/a admin fec state is auto, oper fec state is off Last link flapped 05:21:52 Last clearing of "show interface" counters never 5 interface resets Load-Interval #1: 30 seconds 30 seconds input rate 48 bits/sec, 0 packets/sec 30 seconds output rate 29480 bits/sec, 50 packets/sec input rate 48 bps, 0 pps; output rate 29.48 Kbps, 50 pps Load-Interval #2: 5 minute (300 seconds) 300 seconds input rate 32 bits/sec, 0 packets/sec 300 seconds output rate 29360 bits/sec, 49 packets/sec input rate 32 bps, 0 pps; output rate 29.36 Kbps, 49 pps RX 3795925 unicast packets 2832572 multicast packets 4328 broadcast packets 6632825 input packets 4218958407 bytes 2039753 jumbo packets 0 storm suppression packets 0 runts 0 giants 0 CRC 0 no buffer 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 10 Rx pause TX 8304992 unicast packets 75937156 multicast packets 814 broadcast packets 84242962 output packets 12035830617 bytes 1601477 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause
Please, help me....
Thank you
-
@prx said in Speed Interface error in QinQ interface:
as you can see from the attachment, I disabled:
For some reason, this often doesn’t work the way we would expect
therefore, I would not rely on chance and use loader.conf.local
(you can also see that the box is checked and the feature is still active)like... (loader.conf.local also protects against OP system (FW) changes!):
about the 10 Gig:
Perform a test related to speed and set the switch to fixed speed negotion...
let's see what happensbtw:
with a completely similar problem and we've been talking about this in the forum here for about 6 months now with a college student from the US
I can't find the posts, plus there was a positive result, so it would be worth looking for....One more thing, what CPU you use and what MSI-X
what does this show:dmesg | grep -i msi
-
@DaddyGo said in Speed Interface error in QinQ interface:
dmesg | grep -i msi
Hi,
tomorrow I will modify loader.conf.local and I will configure PfSense and switch ports with fixed speed negotion.
My CPU is AMD EPYC 7272 12-Core and the output of dmesg | grep -i msi is:
xhci0: Unable to map MSI-X table xhci1: Unable to map MSI-X table bnxt0: msix_init qsets capped at 241 bnxt0: Using MSI-X interrupts with 2 vectors bnxt1: msix_init qsets capped at 241 bnxt1: Using MSI-X interrupts with 2 vectors t5nex0: PCIe gen3 x8, 4 ports, 42 MSI-X interrupts, 140 eq, 41 iq xhci2: Unable to map MSI-X table t5nex1: PCIe gen3 x8, 4 ports, 42 MSI-X interrupts, 140 eq, 41 iq xhci3: Unable to map MSI-X table
Thank you very much
-
@prx said in Speed Interface error in QinQ interface:
AMD EPYC 7272
power machine...
@prx "tomorrow I will modify"
okay let's see what happensbtw:
I'm running on Epyc 3151 (Supermicro) with I350-T4 and I350-F4 (8 interfaces) it takes a long time for a well-tuned NIC setupAMD is currently a good choice for a firewall...
a little plus for loader.conf.local...
Allows NIC to process packets as fast as they are received
hw.igb.enable_msix=1
hw.pci.enable_msix=1(of course don't put it on "igb")
-
Hi DaddyGo,
I followed your advice and now my /boot/loader.conf is:
if_bnxt_load="YES" net.inet.tcp.tso=0 net.inet.tcp.lro=0 hw.cxl.enable_msix=1 hw.pci.enable_msix=1 dev.cxl.0.fc=0 dev.cxl.1.fc=0 dev.cxl.2.fc=0 dev.cxl.3.fc=0 dev.cxl.4.fc=0 dev.cxl.5.fc=0 dev.cxl.6.fc=0 dev.cxl.7.fc=0 dev.cxl.0.eee_disabled=1 dev.cxl.1.eee_disabled=1 dev.cxl.2.eee_disabled=1 dev.cxl.3.eee_disabled=1 dev.cxl.4.eee_disabled=1 dev.cxl.5.eee_disabled=1 dev.cxl.6.eee_disabled=1 dev.cxl.7.eee_disabled=1
Then I have set manually, in the GUI, the speed interface on my QinQ interface to 10GBase-T <full-duplex>. The only thing I couldn't do was set up the the speed of my lagg (0,1,2) interfaces to 10G full-duplex. In the GUI under Assignments -> Interfaces I can only select "autoselect" and this is what I see in the dashboard. For example this is the output of ifconfig of the lagg2 (only media information...)
cxl2: media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> cxl6: media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause> lagg2: media: Ethernet autoselect lagg2.100: media: Ethernet autoselect lagg2.100.20: media: Ethernet 10Gbase-T <full-duplex>
I was able only to force media on the lagg2.100.20 interface but not to its parents interfaces...
Thank you
-
@prx said in Speed Interface error in QinQ interface:
In the GUI under Assignments -> Interfaces I can only select "autoselect"
Due to the nature of "LAG", both sides require the same - next:
-Be of the same speed
-Have the same Duplex settingsWhen negotiating the LAG speed (merged), only "auto" can be considered, because of the above.
Just think along, two parent interface, which is 1Gig
in LAG (with LACP) 2 Gigif one of the parent interfaces becomes detached, the combined speed will again be only 1 Gig, so the speed of the LAG interface cannot be predefined, but depends on the group members (from their number / speed)