10G Odyssey
-
Hi folks,
I am trying since a few weeks to upgrade my Pfsense box to 10G. Before buying adapters, I looked up the support in the freeBSD documentation. I am running Pfsense 2.4.5-RELEASE-p1.
When I used an HP NC523, based on the qle3242, I got a link, the interface went up, the packets would be forwarded. Managing the box through the 10G interface was possible.
BUT. No DHCP responses were sent. None at all, even when I just switched the interface on the console menu - without touching the DHCP config.
Plus when I rebooted, the driver would throw errors that basically froze the whole box.
Next try. Intel X520, based on the 82599 Intel chipset. Card is recognized as ix0. But whatever I do, no carrier. Changed the card to another 82599 based card, same story recognized as Intel card - no carrier. I tried generic FS.com SFP+s and DAC, I tried Intel compatible FS.com SFP+s. No luck. (the HP card was running fine with fs.com generic DAC).
Pfsense runs on a HP Microserver G8, I use Mikrotik CSS610-8G-2S+IN and CRS309-1G-8S+IN switches, both on the latest release of SwitchOS.
What am I missing?
TIA,
-V. -
Try running at the command line
ifconfig -vvvvvvm ix0
Does it show the SFP module or DAC as present?
What are you connecting it to?
Steve
-
It says the SFP is present:
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:67:aa hwaddr 64:9d:99:b1:67:aa inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 tentative scopeid 0x1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: FS PN: SFP-10GSR-85 SN: F2020575588 DATE: 2020-07-14 Class: 10G Base-SR Length: short distance Tech: (null) Media: Multimode, 50um Speed: (null) module temperature: 28.56 C Voltage: 3.31 Volts RX: 0.54 mW (-2.64 dBm) TX: 0.00 mW (-40.00 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 53 46 50 2D 31 30 47 53 52 2D 38 35 20 20 20 20 20 20 20 20 03 52 00 AF 00 1A 0A 58 46 32 30 32 30 35 37 35 35 38 38 20 20 20 20 20 32 30 30 37 31 34 20 20 68 F0 03 35 00 00 08 80 6D A0 B8 90 B4 AB F0 FB 5E 05 4C D7 F8 01 CA 00 00 00 00 00 00 00 00 00 8A 8B 53 70
The card is connected via 3m OC4 to the SFP on the Switch.
The switch recognizes the SFP.
-
It looks to be logically down.
It's not transmitting:
TX: 0.00 mW (-40.00 dBm)
Try running
ifconfig ix0 up
then rechecking. Or assigning and enabling the interface in pfSense.Steve
-
No change after I configure it as OPT1 in Pfsense.
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:67:aa hwaddr 64:9d:99:b1:67:aa inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1 inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: FS PN: SFP-10GSR-85 SN: F2020575588 DATE: 2020-07-14 Class: 10G Base-SR Length: short distance Tech: (null) Media: Multimode, 50um Speed: (null) module temperature: 30.99 C Voltage: 3.31 Volts RX: 0.54 mW (-2.64 dBm) TX: 0.68 mW (-1.62 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 53 46 50 2D 31 30 47 53 52 2D 38 35 20 20 20 20 20 20 20 20 03 52 00 AF 00 1A 0A 58 46 32 30 32 30 35 37 35 35 38 38 20 20 20 20 20 32 30 30 37 31 34 20 20 68 F0 03 35 00 00 08 80 6D A0 B8 90 B4 AB F0 FB 5E 05 4C D7
-
@voidrunner said in 10G Odyssey:
TX: 0.68 mW (-1.62 dBm)
Except that it is now actually transmitting. You might try reconnecting the fibre and/or the SFP module.
Does the switch show the link as UP? Do you see any link LEDs?
An x520 NIC would usually 'just work' unless the remote side is set to 1G or otherwise not set to autonegotiate the speed.
Steve
-
@stephenw10 said in 10G Odyssey:
@voidrunner said in 10G Odyssey:
TX: 0.68 mW (-1.62 dBm)
Except that it is now actually transmitting. You might try reconnecting the fibre and/or the SFP module.
Does the switch show the link as UP? Do you see any link LEDs?
An x520 NIC would usually 'just work' unless the remote side is set to 1G or otherwise not set to autonegotiate the speed.
Steve
I've tried using an intel compatible DAC - still no carrier
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:67:aa hwaddr 64:9d:99:b1:67:aa inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1 inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail) vendor: FS PN: SFPP-PC01 SN: S1910031607-1 DATE: 2019-10-24 Class: 1X Copper Passive Length: (null) Tech: Passive Cable Media: Twin Axial Pair Speed: 100 MBytes/sec SFF8472 DUMP (0xA0 0..127 range): 03 04 21 01 00 00 04 00 04 80 D5 00 67 00 00 00 00 00 01 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 53 46 50 50 2D 50 43 30 31 20 20 20 20 20 20 20 44 20 20 20 0C 00 00 31 00 12 00 31 53 31 39 31 30 30 33 31 36 30 37 2D 31 20 20 20 31 39 31 30 32 34 20 20 00 00 00 C1 00 D8 20 20 20 20 20 20 66 61 20 20 20 20 20 20 20 20 20 20 FF FF FF FF FF FF FF FF FF FF FF FF
-
Switch shows recognizes the DAC and shows the link to be up.
-
Is it hard set to 10G at the switch? You need to match both ends so if it try either setting it to auto in the switch or setting the speed to 10G fixed in pfSense.
Steve
-
Actually it's autoselect on the Switch.
I even tried setting the interface on PfSense to 10GB SR
But the adapter seems to remain in auto select
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
ether 64:9d:99:b1:67:aa
hwaddr 64:9d:99:b1:67:aa
inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1
inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
supported media:
media autoselect
media 10Gbase-SR
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FS PN: SFP-10GSR-85 SN: F2020521292 DATE: 2020-07-15
Class: 10G Base-SR
Length: short distance
Tech: (null)
Media: Multimode, 50um
Speed: (null)
module temperature: 37.13 C Voltage: 3.33 Volts
RX: 0.69 mW (-1.61 dBm) TX: 0.67 mW (-1.72 dBm)SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 53 52 2D 38 35 20 20 20 20 41 20 20 20 03 52 00 0C 00 1A 00 00 46 32 30 32 30 35 32 31 32 39 32 20 20 20 20 20 32 30 30 37 31 35 20 20 68 90 01 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-
Update - the problem was with the Mikrotik CSS610-8G-2S+. When switching with the very same config to the CRS309-1G-8S+IN, the links comes up fine.
-
Ah, nice catch! Weird though.