baby jumbo frame WAN MTU problem

JKnott

Have you tried the MTU box on the WAN interface? I don't have PPPoE here, so I can't try it, but it does work with regular Ethernet connections.

stephenw10

Mmm, what happens if you set the Modem Access NIC to 1508 and allow the PPPoE session to follow it?

Do you know the Draytek supports it? I imagine it would though.

Steve

bigsy

@purleigh

All you should have to do is set the MTU to 1500 on WAN (pppoe0). The underlying 'ModemAccess' interface should auto increment its MTU to 1508.

This works for me on a variety of hardware (SG-1100/2100 and Intel i210) but I don't think I've used a Realtek interface for this before.

purleigh

@stephenw10 said in baby jumbo frame WAN MTU problem:

Do you know the Draytek supports it? I imagine it would though.

Yes the DrayTek Vigor 130 modem's ethernet port supports a maximum MTU of 1520
I currently have the modem ethernet port MTU set to 1508

purleigh

Thanks for the suggestions, there were a few combinations that I hadn't previously considered !.

After testing the settings permutations detailed below I still haven't found the combination that reliably allows ModemAccess and simultaneous WAN MTU at 1500.

With WAN[MTU=auto] and ModemAccess[MTU=auto] then I have an always working system BUT with the WAN MTU being standard 1492 for PPPoE (i.e. 1500 - 8 byte overhead).

With WAN[MTU=1500 or 1508] and ModemAccess[MTU=auto] I can achieve the desired outcome (i.e. WAN MTU=1500 and working ModemAccess) BUT only after rebooting pfSense. However if subsequently the PPPoE connection ever needs to renegotiate, perhaps after DSL line noise, DLM forcing a resync, or manually doing [Disconnect WAN] then [Connect WAN], then both WAN and ModemAccess interfaces get stuck indefinitely in alternately failing. Not ideal !

It seems to be an issue of having both PPPoE and ModemAccess simultaneously doing MTU discovery on the same physical NIC on a running system, but is somehow resolved (once) when the same configuration is used on a pfSense reboot.

In the following tables, when I say "Alternating fail", that means that every few seconds the Status>Interface page is toggling between
[ WAN=up, ModemAccess="no carrier" ]
and
[ WAN=down, ModemAccess=up ]

Using the default auto settings (i.e. MTU and MSS fields left blank) -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = auto     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1500  Status=up
-------------------------------------------------------------------------------

Using @bigsy suggestion of only setting WAN MTU to 1500

It works after a pfSense reboot -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1500     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1508  Status=up
-------------------------------------------------------------------------------

BUT fails after a WAN pppoe [Disconnect WAN] then [Connect WAN] -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1500     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

Similarly setting the WAN MTU to 1508 and doing a pfSense reboot -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=up
-------------------------------------------------------------------------------

BUT again fails after a WAN pppoe [Disconnect WAN] then [Connect WAN] -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=Alternating fail
-------------------------------------------------------------------------------

With the ModemAccess "Enable interface" unticked

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=disabled
-------------------------------------------------------------------------------

Using @stephenw10 suggestion "what happens if you set the Modem Access NIC to 1508 and allow the PPPoE session to follow it?"

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = auto     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = 1508     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

Other permutations tried -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = 1508     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = 1500     |  MTU:1500  Status=up
-------------------------------------------------------------------------------

Same as above but with ModemAccess "Enable interface" unticked

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = 1500     |  MTU:1500  Status=disabled
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1516     |  MTU:1492  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1524  Status=Alternating fail
-------------------------------------------------------------------------------

stephenw10

@purleigh said in baby jumbo frame WAN MTU problem:

GA-J3455N-D3H

Both ports on that are the same Realtek controller? They behave the same way?

Steve

purleigh

@stephenw10 said in baby jumbo frame WAN MTU problem:

Both ports on that are the same Realtek controller? They behave the same way?

The pfSense "OS Boot" System Log entries for re0 and re1 look to be the same type of port, so I assume that they work the same.

WAN/ModemAccess port (re0) -

re0: netmap queues/slots: TX 1/256, RX 1/256
re0: Ethernet address: e0:d5:5e:53:60:83
re0: Using defaults for TSO: 65518/35/2048
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
miibus0: <MII bus> on re0
re0: MAC rev. 0x00000000
re0: Chip rev. 0x4c000000
re0: ASPM disabled
re0: Using 1 MSI-X message
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0x91204000-0x91204fff,0x91200000-0x91203fff irq 20 at device 0.0 on pci3
pci3: <ACPI PCI bus> on pcib3
pcib3: <ACPI PCI-PCI bridge> irq 20 at device 19.2 on pci0

LAN port (re1) -

re1: netmap queues/slots: TX 1/256, RX 1/256
re1: Ethernet address: e0:d5:5e:53:60:84
re1: Using defaults for TSO: 65518/35/2048
rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus1
miibus1: <MII bus> on re1
re1: MAC rev. 0x00000000
re1: Chip rev. 0x4c000000
re1: ASPM disabled
re1: Using 1 MSI-X message
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 21 at device 0.0 on pci4
pci4: <ACPI PCI bus> on pcib4
pcib4: <ACPI PCI-PCI bridge> irq 21 at device 19.3 on pci0

stephenw10

Mmm, I wouldn't expect to see any difference there.

purleigh

Does anyone know what the following entry in the pfSense log file means ? -

php-fpm 	54479 	/rc.linkup: Hotplug event detected for MODEMACCESS(opt2) static IP (192.168.xxx.yyy)

What seems to be happening, is that whenever the MTU of ModemAccess(opt2,re0) is greater than 1500, immediately after a new WAN PPPoE session has been successfully established (regardless of WAN PPPoE MTU) then the "Hotplug event detected for ModemAccess" occurs, which then triggers the , "re0" interface to shutdown - taking down both the ModemAccess and the WAN PPPoE interfaces. A few seconds later "kernel re0: link state changed to UP" occurs, a new PPPoE establishment sequence begins, and the whole cycle repeats itself.

This cyclic failing with associated "Hotplug event" ONLY occurs after a successful new PPPoE sessions is established - for example after pressing the [Connect WAN] button.

On the SAME MTU settings, this "Hotplug event" DOES NOT occur on the successful first PPPoE session after a pfSense reboot.

The "Hotplug event" doesn't happen when the PPPoE session (previously established by a reset) is ended by pressing the [Disconnect WAN] button.

Any ideas on why the "Hotplug event" occurs, and how do I stop it ?

stephenw10

Hmm. It implies pfSense thinks the interfaces has been unplugged/plugged.
But I assume you are not seeing other log entries that show it actually disconnected like?:

Aug 17 20:55:27 	check_reload_status 	549 	Linkup starting re0
Aug 17 20:55:27 	kernel 		re0: link state changed to UP
Aug 17 20:55:28 	php-fpm 	512 	/rc.linkup: Hotplug event detected for OPT1(opt1) static IP (10.200.200.1 )
Aug 17 20:55:28 	check_reload_status 	549 	rc.newwanip starting re0
Aug 17 20:55:28 	check_reload_status 	549 	Reloading filter

Steve

purleigh

I finally solved this problem - so thought I'd update this topic just in case it helps someone else !

The unstable WAN MTU problem persisted through a number of pfSense upgrades, and was still an issue on version 2.7.2-RELEASE.

The thing that fixed it was to install the Realtek drivers using -

pkg install -y realtek-re-kmod

which installed -

realtek-re-kmod-198.00_3       Kernel driver for Realtek PCIe Ethernet Controllers

and then setting the WAN MTU to 1508 (including +8 bytes for PPP overhead).

After a day of experimenting, the system seems stable, and the WAN interface MTU is 1500