Supermicro AOC-STGN-I2S - 1Gbps support?

moosport

I'm testing a new pfsense build using Supermicro AOC-STGN-I2S dual port.
SFP+ Port 1 is connected to switch on SFP port. Status for OPT1 is showing no carrier.
Tried using both 10Gbps DAC and 1Gbps transceiver. Locking the port at the switch to 1Gbps.
Unable to set the port on the card to 1Gbps.
Has anyone solved this issue before?

ifconfig -vvvm ix1

media: Ethernet autoselect
    status: no carrier
    supported media:
        media autoselect
        media 10Gbase-SR


ifconfig ix1 media 1000baseTX

ifconfig: SIOCSIFMEDIA (media): Device not configured

stephenw10

That's with the DAC inserted I assume?
That's quite common for DAC modules, they support only 10G or auto. You cannot force them to 1G.
I would expect a 1G SFP fiber module work though. Does that show other supported media types?

moosport

@stephenw10
The results are from using 1Gbps transceiver at Supermicro AOC-STGN-I2S. At the switch, the port is reporting its status. Pfsense is not detecting the signal. Pfsense is not detecting the signal. No other media types report. I have a single port Mellanox ConnectX-3 which I can swap out to test.

status .

stephenw10

Does it see the module correctly in the verbose ifconfig output.

moosport

@stephenw10 ```
Here is the results of ifconfig. It only shows the 2 active interface. WAN and LAN.

[2.6.0-RELEASE][pfSense.local]/root: ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:1f:81:66
	inet6 fe80::ec4:7aff:fe1f:8166%ix0 prefixlen 64 scopeid 0x1
	inet6 fe80::1:1%ix0 prefixlen 64 scopeid 0x1
	inet6 2603:8001:9301:d13e:ec4:7aff:fe1f:8166 prefixlen 64
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT1
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:1f:81:67
	inet6 fe80::ec4:7aff:fe1f:8167%ix1 prefixlen 64 scopeid 0x2
	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether f8:75:a4:88:a7:b3
	inet6 fe80::fa75:a4ff:fe88:a7b3%em0 prefixlen 64 scopeid 0x3
	inet6 2605:e000:7fc0:71:a0a1:eb1d:67ad:b073 prefixlen 128
	inet 76.91.240.192 netmask 0xfffff000 broadcast 255.255.255.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync

Output of each 10Gbps interface.

[2.6.0-RELEASE][pfSense.local]/root:ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:1f:81:66
inet6 fe80::ec4:7aff:fe1f:8166%ix0 prefixlen 64 scopeid 0x1
inet6 fe80::1:1%ix0 prefixlen 64 scopeid 0x1
inet6 2603:8001:9301:d13e:ec4:7aff:fe1f:8166 prefixlen 64
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[2.6.0-RELEASE][pfSense.local]/root: ifconfig ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: OPT1
options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:1f:81:67
inet6 fe80::ec4:7aff:fe1f:8167%ix1 prefixlen 64 scopeid 0x2
inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Sorry, we need the verbose output ifconfig -vvv to see the module info really.

What we can see there looks as expected though.

moosport

@stephenw10
Here is the verbose output.

ifconfig -vvv
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:1f:81:66
	inet6 fe80::ec4:7aff:fe1f:8166%ix0 prefixlen 64 scopeid 0x1
	inet6 fe80::1:1%ix0 prefixlen 64 scopeid 0x1
	inet6 2603:8001:9301:d13e:ec4:7aff:fe1f:8166 prefixlen 64
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-H10GB-CU1M SN: CSC200902630153 DATE: 2020-09-23

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00 
	00 00 01 00 4F 45 4D 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 40 20 53 46 50 2D 48 31 30 47 
	42 2D 43 55 31 4D 20 20 52 20 20 20 01 00 00 F3 
	00 00 00 00 43 53 43 32 30 30 39 30 32 36 33 30 
	31 35 33 20 32 30 30 39 32 33 20 20 00 00 00 C8 
	80 00 11 C6 7E 6C 05 70 09 C4 18 34 1E E2 F0 8B 
	82 4A 8E 00 00 00 00 00 00 00 00 00 15 84 F5 72 
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT1
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:1f:81:67
	inet6 fe80::ec4:7aff:fe1f:8167%ix1 prefixlen 64 scopeid 0x2
	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: UBNT PN: UF-MM-1G SN: FT19110214937 DATE: 2019-11-02
	module temperature: 27.18 C Voltage: 3.23 Volts
	RX: 0.37 mW (-4.27 dBm) TX: 0.29 mW (-5.34 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 00 40 40 0C 55 06 67 00 00 00 
	37 37 00 00 55 42 4E 54 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 00 00 55 46 2D 4D 4D 2D 31 47 
	20 20 20 20 20 20 20 20 20 20 20 20 03 52 00 6F 
	00 0A 00 00 46 54 31 39 31 31 30 32 31 34 39 33 
	37 20 20 20 31 39 31 31 30 32 00 00 68 90 01 61 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether f8:75:a4:88:a7:b3
	inet6 fe80::fa75:a4ff:fe88:a7b3%em0 prefixlen 64 scopeid 0x3
	inet6 2605:e000:7fc0:71:a0a1:eb1d:67ad:b073 prefixlen 128
	inet 76.91.240.192 netmask 0xfffff000 broadcast 255.255.255.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync

stephenw10

Hmm, and that doesn't offer a 1G fixed mode if you run ifconfig -vvvm ix1 ?

moosport

@stephenw10
Here is the output. I have a spare Mellanox ConnectX-3 which I test if this card do not support it.

[2.6.0-RELEASE][pfSense.local]/root: ifconfig -vvm ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT1
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:1f:81:67
	inet6 fe80::ec4:7aff:fe1f:8167%ix1 prefixlen 64 scopeid 0x2
	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-SR
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: UBNT PN: UF-MM-1G SN: FT19110214937 DATE: 2019-11-02
	module temperature: 27.18 C Voltage: 3.23 Volts
	RX: 0.37 mW (-4.27 dBm) TX: 0.28 mW (-5.41 dBm)

stephenw10

Hmm, yeah something weird happening there. That's not a 10G module, it shouldn't be offering 10Gbase-SR as a link type.

moosport

@stephenw10
This is the modules installed. It worked fine for SFP ports.

[Unifi 1Gbps modules ](Ubiquiti U Fiber Multi-Mode SFP... https://www.amazon.com/dp/B01MY5498F?ref=ppx_pop_mob_ap_share)

stephenw10

Right, it's a 1G module but the NIC sees it as 10G and tries to use it as such for some reason.

I haven't seen that before, I'm not sure what could be done there.

Try it i the Mellanox NIC and see what it reports there. Though I will say those ConnectX-3 NICs have always behaved oddly for me.

Steve

moosport

@stephenw10

Tested using Ubuntu Live ISO. Initially both DAC and Unifi module did not work.

So I swap out the Unifi module of the router with a spare and it works immediately.

stephenw10

But still not in pfSense?

moosport

@stephenw10
Pfsense sees it now. Sorry I forget to mentioned it.

Now I’m trying to figure out how to make wireless devices see the printer, unifi cloud key and HomeKit hub from the other subnet.

stephenw10

The best way to do that is to just use the IP directly. However I imagine you might have some devices that need to 'discover' those resources and have no option to enter an IP directly. If that's the case you might need something to proxy that discovery traffic like Avahi.

Steve

moosport

@stephenw10
Will give it a try when I get some time. For now, first step of moving pfsense to bare metal is done.

Thanks Stephen!