Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [BUG?] New PPPoE module (if_pppoe) causes high "Errors Out" on WAN (Vivo Fibra)

    Scheduled Pinned Locked Moved Routing and Multi WAN
    47 Posts 8 Posters 3.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      OK if you can try running this at the actual CLI (not the gui command prompt!) and wait for some errors to occur:
      dtrace -n 'fbt::if_inc_counter:entry / arg1 == 3 / { stack(); }'

      I managed to trigger some errors in my local test connection by restarting the server but you should see some pretty quickly with the rate you're seeing.

      It should look something like this:

      2.8.0-RELEASE][admin@m370.stevew.lan]/root: dtrace -n 'fbt::if_inc_counter:entry / arg1 == 3 / { stack(); }'
      dtrace: description 'fbt::if_inc_counter:entry ' matched 1 probe
      
      CPU     ID                    FUNCTION:NAME
        0  43915             if_inc_counter:entry 
                    if_pppoe.ko`sppp_output+0x15e
                    kernel`pf_route+0x8e5
                    kernel`pf_test+0x1093
                    kernel`pf_check_out+0x2e
                    kernel`pfil_mbuf_out+0x38
                    kernel`ip_output+0xbea
                    kernel`rip_send+0x3f4
                    kernel`sosend_generic_locked+0x60f
                    kernel`sosend_generic+0x61
                    kernel`sousrsend+0x5f
                    kernel`kern_sendit+0x144
                    kernel`sendit+0x1a8
                    kernel`sys_sendto+0x4d
                    kernel`amd64_syscall+0x115
                    kernel`0xffffffff812c145b
      

      Use ctl+c to esacpe once you have some traces.

      brookheatherB 1 Reply Last reply Reply Quote 0
      • brookheatherB
        brookheather @stephenw10
        last edited by

        @stephenw10 here you go:

        [2.8.0-RELEASE][admin@pfSense.localdomain]/root: dtrace -n 'fbt::if_inc_counter:entry / arg1 == 3 / { stack(); }'
        dtrace: description 'fbt::if_inc_counter:entry ' matched 1 probe
        CPU     ID                    FUNCTION:NAME
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          3  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          1  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          1  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          1  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`swi_net+0x128
                      kernel`ithread_loop+0x239
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
          0  43915             if_inc_counter:entry 
                      if_pppoe.ko`sppp_output+0x1ec
                      kernel`pf_route+0x8e5
                      kernel`pf_test+0x1093
                      kernel`pf_check_out+0x2e
                      kernel`pfil_mbuf_fwd+0x38
                      kernel`ip_tryforward+0x288
                      kernel`ip_input+0x34a
                      kernel`netisr_dispatch_src+0x1fa
                      kernel`ether_demux+0x194
                      kernel`ether_nh_input+0x32c
                      kernel`netisr_dispatch_src+0x9f
                      kernel`ether_input+0x56
                      kernel`iflib_rxeof+0xbde
                      kernel`_task_fn_rx+0x72
                      kernel`gtaskqueue_run_locked+0x14e
                      kernel`gtaskqueue_thread_loop+0xc2
                      kernel`fork_exit+0x7b
                      kernel`0xffffffff812c1bae
        
        stephenw10S 1 Reply Last reply Reply Quote 1
        • D
          dorabiatto
          last edited by

          [EN_US]

          Just wanted to share my experience that might help others dealing with the same issue described in this thread.

          I was getting a lot of “Errors Out” on the WAN interface when using pfSense with PPPoE on Vivo Fibra, specifically with the Mitrastar ONU provided by the ISP. I tried changing MTU, tuning system settings, updating pfSense — nothing helped. The errors kept increasing, affecting connection stability and performance.

          I decided to replace the Mitrastar ONU with a VSOL V2802RH ONU set to bridge mode — and that completely stopped the errors. Since the switch, pfSense shows at most 1 single error out right after boot, and then nothing more.

          Summary:

          Before (Mitrastar): constant Errors Out, especially under load.

          After (VSOL): only 1 error after boot, then stable with no further errors.

          So it seems the issue is likely related to some kind of compatibility quirk between the Mitrastar ONU and the new if_pppoe module. Switching to a different ONU (VSOL) significantly reduces or eliminates the issue.

          [PT-BR]

          Fica aqui meu relato que pode ajudar outros usuários enfrentando o mesmo problema descrito neste tópico.

          Estava tendo altos “Errors Out” na interface WAN ao usar pfSense com PPPoE da Vivo Fibra, especialmente com a ONU Mitrastar fornecida pela operadora. Nada resolvia: MTU ajustado, tunable, atualizações — e os erros persistiam, afetando a estabilidade e performance.

          Resolvi trocar a ONU por uma VSOL XPON configurada em modo bridge — e os erros pararam completamente. Desde a troca, o pfSense mostra no máximo 1 erro Out isolado após boot, e depois disso a conexão segue estável, sem erros acumulando.

          Resumo:

          Antes (Mitrastar): errors out constantes, especialmente com alto tráfego.

          Depois (VSOL): apenas 1 erro após boot, e mais nada.

          Ou seja, o problema parece estar relacionado à compatibilidade (ou comportamento estranho) da Mitrastar com o módulo PPPoE novo (if_pppoe), e a troca de ONU resolve ou mitiga o problema drasticamente.

          8ad45427-1bee-49c2-8fdd-678055d828b0-image.png

          1 Reply Last reply Reply Quote 1
          • stephenw10S
            stephenw10 Netgate Administrator @brookheather
            last edited by

            @brookheather Did the PPPoE remain up during that time?

            brookheatherB 1 Reply Last reply Reply Quote 0
            • brookheatherB
              brookheather @stephenw10
              last edited by

              @stephenw10 Yes those errors are beig produced while the PPPoE connection is up and running.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                OK try:
                dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }'

                If that shows ENOBUFS (55) then it looks like it's exhausting the available buffers on your system for some reason. We can try just increasing it to allow for larger bursts.

                brookheatherB 1 Reply Last reply Reply Quote 0
                • brookheatherB
                  brookheather @stephenw10
                  last edited by

                  @stephenw10 here is the output:

                  [2.8.0-RELEASE][admin@pfSense.localdomain]/root: dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }'
                  dtrace: description 'fbt::if_sendq_enqueue:return ' matched 1 probe
                  CPU     ID                    FUNCTION:NAME
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                    1  65524          if_sendq_enqueue:return 
                                if_pppoe.ko`sppp_output+0x1d0
                                kernel`pf_route+0x8e5
                                kernel`pf_test+0x1093
                                kernel`pf_check_out+0x2e
                                kernel`pfil_mbuf_fwd+0x38
                                kernel`ip_tryforward+0x288
                                kernel`ip_input+0x34a
                                kernel`swi_net+0x128
                                kernel`ithread_loop+0x239
                                kernel`fork_exit+0x7b
                                kernel`0xffffffff812c1bae
                  => 55
                  
                  
                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by stephenw10

                    Ok great so it is exhausting the interface buffer.

                    Check the current buffer size: sysctl net.link.ifqmaxlen

                    Try increasing it by 50%. It's a loader value so create the file /boot/loader.conf.local and add the line:
                    net.link.ifqmaxlen=192

                    Then reboot. And see if that makes any difference.

                    brookheatherB 1 Reply Last reply Reply Quote 0
                    • brookheatherB
                      brookheather @stephenw10
                      last edited by brookheather

                      @stephenw10 the current size is 128 - I will apply the change and reboot - might be tomorrow now. Why would this only be affecting the new if_pppoe code - how does it differ from the old code? Is it really an error if the buffer is full - doesn't it just wait until it can write the packet?

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        The if_pppoe code is very different so there could be any number of reasons.

                        Another interesting question might be why it's only affecting a small number of users. Something unique about your connection maybe.

                        brookheatherB 1 Reply Last reply Reply Quote 0
                        • brookheatherB
                          brookheather @stephenw10
                          last edited by

                          @stephenw10 I increased net.link.ifqmaxlen to 192 and rebooted - checked this was set correctly but still getting WAN out errors - it doesn't seem to have made any difference.

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Hmm, actually at the same rate? Not reduced at all?

                            brookheatherB 1 Reply Last reply Reply Quote 0
                            • brookheatherB
                              brookheather @stephenw10
                              last edited by

                              @stephenw10 if anything higher:

                              In/out packets 946815/304707 (1.15 GiB/161.23 MiB)
                              In/out packets (pass) 946815/304707 (1.15 GiB/161.23 MiB)
                              In/out packets (block) 770/0 (69 KiB/0 B)
                              In/out errors 0/3250
                              Collisions 0

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                OK try setting that to something much larger like 1024. If it still throws errors we know it's not simply bursting up to the buffer.

                                You could also try checking the error type again with:
                                dtrace -n 'fbt::if_sendq_enqueue:return / arg1 != 0 / { stack(); printf("=> %d", arg1); }'

                                Make sure it's still throwing error 55 with the buffer at 192B

                                brookheatherB 2 Replies Last reply Reply Quote 0
                                • brookheatherB
                                  brookheather @stephenw10
                                  last edited by

                                  @stephenw10 still throwing error 55 with buffer set to 192 - will try 1024 later.

                                  1 Reply Last reply Reply Quote 0
                                  • brookheatherB
                                    brookheather @stephenw10
                                    last edited by brookheather

                                    @stephenw10 I increased the setting to 1024, rebooted and still getting loads of errors with code 55.

                                    In/out packets 382831/240827 (445.27 MiB/265.41 MiB)
                                    In/out packets (pass) 382831/240827 (445.27 MiB/265.41 MiB)
                                    In/out packets (block) 1255/0 (168 KiB/0 B)
                                    In/out errors 0/5242
                                    Collisions 0

                                    1 Reply Last reply Reply Quote 2
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Ok, we'll dig deeper.

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        tlex
                                        last edited by

                                        Would you have any VIP on the WAN interface by any chance ?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          There is a patch for that but the symptoms there are generally that the link continually reconnects.

                                          But, seperately, @brookheather are you using ALTQ traffic shaping on the pppoe interface? It could be exhausting the queue on that.

                                          brookheatherB 1 Reply Last reply Reply Quote 0
                                          • brookheatherB
                                            brookheather @stephenw10
                                            last edited by

                                            @stephenw10 Yes I am using the CODELQ traffic shaper on both LAN and WAN. WAN is set to 73Mb/s (for 500/75 connection).

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.