[BUG?] New PPPoE module (if_pppoe) causes high "Errors Out" on WAN (Vivo Fibra)
-
@stephenw10 here you go:
[2.8.0-RELEASE][admin@pfSense.localdomain]/root: dtrace -n 'fbt::if_inc_counter:entry / arg1 == 3 / { stack(); }' dtrace: description 'fbt::if_inc_counter:entry ' matched 1 probe CPU ID FUNCTION:NAME 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 3 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 1 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 1 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 1 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae 0 43915 if_inc_counter:entry if_pppoe.ko`sppp_output+0x1ec kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`netisr_dispatch_src+0x1fa kernel`ether_demux+0x194 kernel`ether_nh_input+0x32c kernel`netisr_dispatch_src+0x9f kernel`ether_input+0x56 kernel`iflib_rxeof+0xbde kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x14e kernel`gtaskqueue_thread_loop+0xc2 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae -
[EN_US]
Just wanted to share my experience that might help others dealing with the same issue described in this thread.
I was getting a lot of “Errors Out” on the WAN interface when using pfSense with PPPoE on Vivo Fibra, specifically with the Mitrastar ONU provided by the ISP. I tried changing MTU, tuning system settings, updating pfSense — nothing helped. The errors kept increasing, affecting connection stability and performance.
I decided to replace the Mitrastar ONU with a VSOL V2802RH ONU set to bridge mode — and that completely stopped the errors. Since the switch, pfSense shows at most 1 single error out right after boot, and then nothing more.
Summary:
Before (Mitrastar): constant Errors Out, especially under load.
After (VSOL): only 1 error after boot, then stable with no further errors.
So it seems the issue is likely related to some kind of compatibility quirk between the Mitrastar ONU and the new if_pppoe module. Switching to a different ONU (VSOL) significantly reduces or eliminates the issue.
[PT-BR]
Fica aqui meu relato que pode ajudar outros usuários enfrentando o mesmo problema descrito neste tópico.
Estava tendo altos “Errors Out” na interface WAN ao usar pfSense com PPPoE da Vivo Fibra, especialmente com a ONU Mitrastar fornecida pela operadora. Nada resolvia: MTU ajustado, tunable, atualizações — e os erros persistiam, afetando a estabilidade e performance.
Resolvi trocar a ONU por uma VSOL XPON configurada em modo bridge — e os erros pararam completamente. Desde a troca, o pfSense mostra no máximo 1 erro Out isolado após boot, e depois disso a conexão segue estável, sem erros acumulando.
Resumo:
Antes (Mitrastar): errors out constantes, especialmente com alto tráfego.
Depois (VSOL): apenas 1 erro após boot, e mais nada.
Ou seja, o problema parece estar relacionado à compatibilidade (ou comportamento estranho) da Mitrastar com o módulo PPPoE novo (if_pppoe), e a troca de ONU resolve ou mitiga o problema drasticamente.
-
@brookheather Did the PPPoE remain up during that time?
-
@stephenw10 Yes those errors are beig produced while the PPPoE connection is up and running.
-
OK try:
dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }'If that shows ENOBUFS (55) then it looks like it's exhausting the available buffers on your system for some reason. We can try just increasing it to allow for larger bursts.
-
@stephenw10 here is the output:
[2.8.0-RELEASE][admin@pfSense.localdomain]/root: dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }' dtrace: description 'fbt::if_sendq_enqueue:return ' matched 1 probe CPU ID FUNCTION:NAME 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 1 65524 if_sendq_enqueue:return if_pppoe.ko`sppp_output+0x1d0 kernel`pf_route+0x8e5 kernel`pf_test+0x1093 kernel`pf_check_out+0x2e kernel`pfil_mbuf_fwd+0x38 kernel`ip_tryforward+0x288 kernel`ip_input+0x34a kernel`swi_net+0x128 kernel`ithread_loop+0x239 kernel`fork_exit+0x7b kernel`0xffffffff812c1bae => 55 -
Ok great so it is exhausting the interface buffer.
Check the current buffer size:
sysctl net.link.ifqmaxlenTry increasing it by 50%. It's a loader value so create the file /boot/loader.conf.local and add the line:
net.link.ifqmaxlen=192Then reboot. And see if that makes any difference.
-
@stephenw10 the current size is 128 - I will apply the change and reboot - might be tomorrow now. Why would this only be affecting the new if_pppoe code - how does it differ from the old code? Is it really an error if the buffer is full - doesn't it just wait until it can write the packet?
-
The if_pppoe code is very different so there could be any number of reasons.
Another interesting question might be why it's only affecting a small number of users. Something unique about your connection maybe.
-
@stephenw10 I increased net.link.ifqmaxlen to 192 and rebooted - checked this was set correctly but still getting WAN out errors - it doesn't seem to have made any difference.
-
Hmm, actually at the same rate? Not reduced at all?
-
@stephenw10 if anything higher:
In/out packets 946815/304707 (1.15 GiB/161.23 MiB)
In/out packets (pass) 946815/304707 (1.15 GiB/161.23 MiB)
In/out packets (block) 770/0 (69 KiB/0 B)
In/out errors 0/3250
Collisions 0 -
OK try setting that to something much larger like 1024. If it still throws errors we know it's not simply bursting up to the buffer.
You could also try checking the error type again with:
dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }'Make sure it's still throwing error 55 with the buffer at 192B
-
@stephenw10 still throwing error 55 with buffer set to 192 - will try 1024 later.
-
@stephenw10 I increased the setting to 1024, rebooted and still getting loads of errors with code 55.
In/out packets 382831/240827 (445.27 MiB/265.41 MiB)
In/out packets (pass) 382831/240827 (445.27 MiB/265.41 MiB)
In/out packets (block) 1255/0 (168 KiB/0 B)
In/out errors 0/5242
Collisions 0 -
Ok, we'll dig deeper.
-
Would you have any VIP on the WAN interface by any chance ?
-
There is a patch for that but the symptoms there are generally that the link continually reconnects.
But, seperately, @brookheather are you using ALTQ traffic shaping on the pppoe interface? It could be exhausting the queue on that.
-
@stephenw10 Yes I am using the CODELQ traffic shaper on both LAN and WAN. WAN is set to 73Mb/s (for 500/75 connection).
-
Aha, and what is the queue length set to on WAN? Try increasing it and see if that reduces the output errors.
