baby jumbo frame WAN MTU problem

purleigh

pfSense version 2.5.2

Motherboard :
Gigabyte GA-J3455N-D3H
BIOS version = F2
CPU = Celeron J3455
Ethernet = Realtek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet

External modem :
DrayTek Vigor 130
Firmware version = 3.8.4.1_BT

What I'm trying to achieve :
A) WAN MTU of 1500 by utilizing baby jumbo frames on pppoe connection
and
B) Access modem GUI from LAN using method described in https://docs.netgate.com/pfsense/en/latest/recipes/modem-access.html

Scenarios that work :

If WAN(pppoe0) and ModemAccess(opt2,re0) interface MTU and MSS settings are left blank,
results in ModemAccess working with MTU=1500 but WAN MTU=1492 due to pppoe 8 byte overhead.
Setting WAN(pppoe0) MTU=1508 MSS=1460 and with ModemAccess(opt2,re0) interface MTU and MSS settings left blank, AND then rebooting pfSense,
results in ModemAccess working (MTU=1516) and WAN(pppoe0) works (MTU=1500)

The problem I'm trying to solve :
In pfSense GUI, if I navigate to Status>Interfaces>WAN-Interface>PPPoE, and then press [Disconnect WAN] later followed by [Connect WAN] to establish a new connection -

In scenario 1) everything works every time, but with the limitation of WAN MTU=1492

In scenario 2) pfSense Status>Dashboard is showing the WAN and ModemAccess interfaces alternately going up and down every few seconds indefinitely (until reboot)

As it looks like interface (re0) is crashing up and down due to conflicting MTU values when the pppoe session is being renegotiated, I've tried setting the ModemAccess MTU to 1500, 1508, and 1516 (rather than leaving it blank for auto configuration) but that made no difference.

The log file shows several instances -
kernel re0: link state changed to DOWN
check_reload_status Linkup starting re0
... ...
kernel re0: link state changed to UP
check_reload_status Linkup starting re0

Any ideas on what settings I need to solve this ?

JKnott

@purleigh said in baby jumbo frame WAN MTU problem:

A) WAN MTU of 1500 by utilizing baby jumbo frames on pppoe connection

Is that even possible. When you set the MTU on an interface, it can be as big as the hardware supports, but with ADSL & PPPoE you're limited to 1492 bytes, by the ISP.

purleigh

Here in the UK, both ADSL and VDSL lines on the British Telecom network, support 1500 MTU.
It definitely works, my line has been that way for at least 15 years

JKnott

@purleigh

Have you tried the MTU box on the WAN interface? I don't have PPPoE here, so I can't try it, but it does work with regular Ethernet connections.

stephenw10

Mmm, what happens if you set the Modem Access NIC to 1508 and allow the PPPoE session to follow it?

Do you know the Draytek supports it? I imagine it would though.

Steve

bigsy

@purleigh

All you should have to do is set the MTU to 1500 on WAN (pppoe0). The underlying 'ModemAccess' interface should auto increment its MTU to 1508.

This works for me on a variety of hardware (SG-1100/2100 and Intel i210) but I don't think I've used a Realtek interface for this before.

purleigh

@stephenw10 said in baby jumbo frame WAN MTU problem:

Do you know the Draytek supports it? I imagine it would though.

Yes the DrayTek Vigor 130 modem's ethernet port supports a maximum MTU of 1520
I currently have the modem ethernet port MTU set to 1508

purleigh

Thanks for the suggestions, there were a few combinations that I hadn't previously considered !.

After testing the settings permutations detailed below I still haven't found the combination that reliably allows ModemAccess and simultaneous WAN MTU at 1500.

With WAN[MTU=auto] and ModemAccess[MTU=auto] then I have an always working system BUT with the WAN MTU being standard 1492 for PPPoE (i.e. 1500 - 8 byte overhead).

With WAN[MTU=1500 or 1508] and ModemAccess[MTU=auto] I can achieve the desired outcome (i.e. WAN MTU=1500 and working ModemAccess) BUT only after rebooting pfSense. However if subsequently the PPPoE connection ever needs to renegotiate, perhaps after DSL line noise, DLM forcing a resync, or manually doing [Disconnect WAN] then [Connect WAN], then both WAN and ModemAccess interfaces get stuck indefinitely in alternately failing. Not ideal !

It seems to be an issue of having both PPPoE and ModemAccess simultaneously doing MTU discovery on the same physical NIC on a running system, but is somehow resolved (once) when the same configuration is used on a pfSense reboot.

In the following tables, when I say "Alternating fail", that means that every few seconds the Status>Interface page is toggling between
[ WAN=up, ModemAccess="no carrier" ]
and
[ WAN=down, ModemAccess=up ]

Using the default auto settings (i.e. MTU and MSS fields left blank) -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = auto     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1500  Status=up
-------------------------------------------------------------------------------

Using @bigsy suggestion of only setting WAN MTU to 1500

It works after a pfSense reboot -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1500     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1508  Status=up
-------------------------------------------------------------------------------

BUT fails after a WAN pppoe [Disconnect WAN] then [Connect WAN] -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1500     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

Similarly setting the WAN MTU to 1508 and doing a pfSense reboot -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=up
-------------------------------------------------------------------------------

BUT again fails after a WAN pppoe [Disconnect WAN] then [Connect WAN] -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=Alternating fail
-------------------------------------------------------------------------------

With the ModemAccess "Enable interface" unticked

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=up
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1516  Status=disabled
-------------------------------------------------------------------------------

Using @stephenw10 suggestion "what happens if you set the Modem Access NIC to 1508 and allow the PPPoE session to follow it?"

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = auto     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = 1508     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

Other permutations tried -

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1500  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = 1508     |  MTU:1508  Status=Alternating fail
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = 1500     |  MTU:1500  Status=up
-------------------------------------------------------------------------------

Same as above but with ModemAccess "Enable interface" unticked

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1508     |  MTU:1492  Status=up
ModemAccess(opt2,re0)  |    MTU = 1500     |  MTU:1500  Status=disabled
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
	Interface      | Interface setting |  Resulting status                 
-----------------------+-------------------+-----------------------------------
        WAN(pppoe0)    |    MTU = 1516     |  MTU:1492  Status=Alternating fail
ModemAccess(opt2,re0)  |    MTU = auto     |  MTU:1524  Status=Alternating fail
-------------------------------------------------------------------------------

stephenw10

@purleigh said in baby jumbo frame WAN MTU problem:

GA-J3455N-D3H

Both ports on that are the same Realtek controller? They behave the same way?

Steve

purleigh

@stephenw10 said in baby jumbo frame WAN MTU problem:

Both ports on that are the same Realtek controller? They behave the same way?

The pfSense "OS Boot" System Log entries for re0 and re1 look to be the same type of port, so I assume that they work the same.

WAN/ModemAccess port (re0) -

re0: netmap queues/slots: TX 1/256, RX 1/256
re0: Ethernet address: e0:d5:5e:53:60:83
re0: Using defaults for TSO: 65518/35/2048
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
miibus0: <MII bus> on re0
re0: MAC rev. 0x00000000
re0: Chip rev. 0x4c000000
re0: ASPM disabled
re0: Using 1 MSI-X message
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0x91204000-0x91204fff,0x91200000-0x91203fff irq 20 at device 0.0 on pci3
pci3: <ACPI PCI bus> on pcib3
pcib3: <ACPI PCI-PCI bridge> irq 20 at device 19.2 on pci0

LAN port (re1) -

re1: netmap queues/slots: TX 1/256, RX 1/256
re1: Ethernet address: e0:d5:5e:53:60:84
re1: Using defaults for TSO: 65518/35/2048
rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus1
miibus1: <MII bus> on re1
re1: MAC rev. 0x00000000
re1: Chip rev. 0x4c000000
re1: ASPM disabled
re1: Using 1 MSI-X message
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 21 at device 0.0 on pci4
pci4: <ACPI PCI bus> on pcib4
pcib4: <ACPI PCI-PCI bridge> irq 21 at device 19.3 on pci0

stephenw10

Mmm, I wouldn't expect to see any difference there.

purleigh

Does anyone know what the following entry in the pfSense log file means ? -

php-fpm 	54479 	/rc.linkup: Hotplug event detected for MODEMACCESS(opt2) static IP (192.168.xxx.yyy)

What seems to be happening, is that whenever the MTU of ModemAccess(opt2,re0) is greater than 1500, immediately after a new WAN PPPoE session has been successfully established (regardless of WAN PPPoE MTU) then the "Hotplug event detected for ModemAccess" occurs, which then triggers the , "re0" interface to shutdown - taking down both the ModemAccess and the WAN PPPoE interfaces. A few seconds later "kernel re0: link state changed to UP" occurs, a new PPPoE establishment sequence begins, and the whole cycle repeats itself.

This cyclic failing with associated "Hotplug event" ONLY occurs after a successful new PPPoE sessions is established - for example after pressing the [Connect WAN] button.

On the SAME MTU settings, this "Hotplug event" DOES NOT occur on the successful first PPPoE session after a pfSense reboot.

The "Hotplug event" doesn't happen when the PPPoE session (previously established by a reset) is ended by pressing the [Disconnect WAN] button.

Any ideas on why the "Hotplug event" occurs, and how do I stop it ?

stephenw10

Hmm. It implies pfSense thinks the interfaces has been unplugged/plugged.
But I assume you are not seeing other log entries that show it actually disconnected like?:

Aug 17 20:55:27 	check_reload_status 	549 	Linkup starting re0
Aug 17 20:55:27 	kernel 		re0: link state changed to UP
Aug 17 20:55:28 	php-fpm 	512 	/rc.linkup: Hotplug event detected for OPT1(opt1) static IP (10.200.200.1 )
Aug 17 20:55:28 	check_reload_status 	549 	rc.newwanip starting re0
Aug 17 20:55:28 	check_reload_status 	549 	Reloading filter

Steve

purleigh

I finally solved this problem - so thought I'd update this topic just in case it helps someone else !

The unstable WAN MTU problem persisted through a number of pfSense upgrades, and was still an issue on version 2.7.2-RELEASE.

The thing that fixed it was to install the Realtek drivers using -

pkg install -y realtek-re-kmod

which installed -

realtek-re-kmod-198.00_3       Kernel driver for Realtek PCIe Ethernet Controllers

and then setting the WAN MTU to 1508 (including +8 bytes for PPP overhead).

After a day of experimenting, the system seems stable, and the WAN interface MTU is 1500