pfsense 2.7.2 - realtek sfp port down after reboot

Domex

Hello I'm very noob
I contigured pfsense on the same PC (Dell wye 5070) in the past without problem.
After a while I resumed using it, meanwhile vendor upgraded the FTTH 2.5 Gb and I updated to 2.7.2
Everything seems went going good till first reboot, The WAN SFP port led doesn't start.
Rebooted and halt the system doesn't solve.
Only solution hot unplug SFP and plug agian.

I saw realtek NIC dirver has some problemin the past so I tried solution "pkg install realtek-re-kmod" used with 2.6.0 (https://forum.netgate.com/topic/98740/any-way-to-install-nano-on-2-3) but doesn't fix the problem.

When the WAN is up I don't see any problem.
Can someone kindly help me?

re1 = WAN

re0@pci0:1:0:0: class=0x020000 rev=0x15 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1028 subdevice=0x080c
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet
re1@pci0:2:0:0: class=0x020000 rev=0x15 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1028 subdevice=0x080c
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

stephenw10

Is it actually loading the alternative driver? Do you see that in the boot logs?

What ifconfig -vvvm re1 show when it's working? When it's not coming up?

Steve

Domex

@stephenw10
hello,
I rebooted a couple of time without problem,
If halt the system and restart face the issue,.

This is the result of "ifconfig -vvvm re1" when pfsense is working

re1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=201b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
capabilities=381b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether 6c:2b:59:64:c0:5f
inet6 fe80::6e2b:59ff:fe64:c05f%re1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

this is after halt the system

re1: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=201b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
capabilities=381b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether 6c:2b:59:64:c0:5f
inet6 fe80::6e2b:59ff:fe64:c05f%re1 prefixlen 64 tentative scopeid 0x2
media: Ethernet autoselect
status: no carrier
supported media:
media autoselect
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

this is part of boot log, maybe the problem?

ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1
random: entropy device external interface
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.

stephenw10

Nope, those are all expected log entries.

So what NIC are you actually using there? What SFP module?

Does it link if you run: ifconfig re1 down; ifconfig re1 up

Or if you run: ifconfig re1 media autoselect

Domex

@stephenw10
this is the module "Genuine Connector SFP I/O Dell OptiPlex Wyse 5070/57GKF 057GKF"

ifconfig re1 down; ifconfig re1 up doesn't solve

ifconfig re1 media autoselect doesn't solve

unlpug the F/O cable doesn't solve

The only solution is hot unplug cable and SPF module

stephenw10

@Domex said in pfsense 2.7.2 - realtek sfp port down after reboot:

Genuine Connector SFP I/O Dell OptiPlex Wyse 5070/57GKF 057GKF

Hmm, OK so something fairly custom then.

What is the module you're using?

Have you tried other modules?

Domex

@stephenw10
The module is DPN-100 (D-Link)
It is provided by FTTH provider.
I can't replace with another one, they register the GPON S/N.

(I have another one, huawei, but I don't know how can I overwrite the SN to bypass their check)

stephenw10

Well I'd want to try the NIC with any other SFP module just to be sure it links correctly at boot. So any 1G SFP module linked to something local.
Those GPON modules can be quirky. But so can Realtek NICs....

Domex

@stephenw10
I tryed with another module, reboted 3 time, all the times, the led starts blinking.

Then I halt 3-4 times with with D-Link module and started succesfully, till suddently again the problem...

It happen frequently but not always...

stephenw10

Hmm, I would probably try to test in some other OS if you can. Make sure it's not a hardware issue.

Though I'm not sure there's much we can do. Did you confirm it's loading the alternate driver? You see the updated driver version in the boot log?

Domex

@stephenw10 said in pfsense 2.7.2 - realtek sfp port down after reboot:

You see the updated driver version in the boot log?

I'm not sure it is mounting the updated version.
just run this command:
pkg install realtek-re-kmod

and added (in bold) 2 rows into the /boot/loader.conf

kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
opensolaris_load="YES"
zfs_load="YES"
kern.geom.label.gptid.enable="0"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
opensolaris_load="YES"
zfs_load="YES"
if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"
boot_serial="NO"
autoboot_delay="3"
hw.hn.vf_transparent="0"
hw.hn.use_if_start="1"
net.link.ifqmaxlen="128"
machdep.hwpstate_pkg_ctrl="1"
net.pf.states_hashsize="524288"

What should I look for boot log?

If I connect the same module on another router Zyxel EX5601 (the one the FTTH vendor gave me), I never see the problem, it always boot and and the wan interface starts successufully

stephenw10

The driver version is printed at boot from the alternative driver like:

[24.03-RELEASE][admin@apu.stevew.lan]/root: dmesg | grep re0
re0: <Realtek PCIe GbE Family Controller> port 0x1000-0x10ff mem 0xf7900000-0xf7900fff,0xf7800000-0xf7803fff irq 16 at device 0.0 on pci1
re0: Using Memory Mapping!
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: version:1.99.04
re0: Ethernet address: 00:0d:b9:33:35:10
re0: Ethernet address: 00:0d:b9:33:35:10

The default driver does not show a version.

Domex

@stephenw10 said in pfsense 2.7.2 - realtek sfp port down after reboot:

dmesg | grep re

this is mine
(dmesg | grep re1) copied the part related re1, my WAN.

re1: <Realtek PCIe GbE Family Controller> port 0xd000-0xd0ff mem 0xa1104000-0xa1104fff,0xa1100000-0xa1103fff at device 0.0 on pci2
re1: Using Memory Mapping!
re1: Using 1 MSI-X message
re1: ASPM disabled
re1: version:1.98.00
re1: Ethernet address: 6c:2b:59:64:c0:5f
This product is covered by one or more of the following patents:
re1: Ethernet address: 6c:2b:59:64:c0:5f

stephenw10

Ok, it is loading the driver so that clearly doesn't help.

Domex

@stephenw10
My driver version is a bit older than yours.
Do you think I can update?

stephenw10

There's a slightly newer driver in Plus but I'd be very surprised if that made any difference on that NIC.
You can see what's been included here:
https://github.com/alexdupre/rtl_bsd_drv/tree/v1.99

Domex

@stephenw10
I did another test:
Halt the system many time and every time also unplugged power cable.
All the time the WAN went up succesfully....

Domex

This post is deleted!

stephenw10

Hmm, so it only fails to reconnect when coming out of standby?

The system could be holding the NIC powered to allow WoL etc. There may be some BIOS settings to disable that.