Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multiple issues, firewall freezes and whole network goes down.

    Scheduled Pinned Locked Moved
    General pfSense Questions
    4
    55
    1.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      slu @Laxarus
      last edited by

      @Laxarus
      we have the same hardware but not the 25 gbps card.

      Please check over the IPMI interface for some PCIe, ... errors, we had a faulty broadcom card some months ago.

      pfSense Gold subscription

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Sometimes ctl+t is the only thing that will produce a response.

        LaxarusL 1 Reply Last reply Reply Quote 0
        • LaxarusL
          Laxarus @stephenw10
          last edited by

          @stephenw10 will try ctrl + c, if the same thing happens again (hoping not), I will try to troubleshoot with WAN when I go back (right now I only have remote access).
          There is only one constant in all the situations, when WAN goes down, there is a big chance of firewall crashing or freezing.
          And the two bugs that you have stated is contributing to this somehow when WAN goes down. Hopefully, the next release of pfsense will take care of these bugs.
          Thanks for bearing with me until now and I really appreciate it.

          @slu thanks for the suggestion. I have checked the maintenance and health logs on the IPMI but there is nothing noteworthy there. It all seems normal.

          1 Reply Last reply Reply Quote 0
          • LaxarusL
            Laxarus
            last edited by

            So, I had the same issue again this morning and I still have no idea why this is happening. @stephenw10 I have tried ctrl + t and no response to that neither.

            Any advise to debugging this is very much appreciated.

            Full log here, the freeze happened around Sep 16 07:00
            system.log.0

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              You need to tune the OVPN_S2S_VPNV4 gateway. It's throwing alarms repeatedly. It's clearly a pretty bad route because the alarms are legitimate for a default settings . However reloading the firewall each tie it fires is not helping anything. You might just disable the monitoring or monitoring action on that gateway.

              But that shouldn't cause it to stop responding. The actual failure appears to happen here:

              Sep 16 07:18:45 FIREWALL rc.gateway_alarm[63113]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:91.226ms RTTsd:79.944ms Loss:21%)
Sep 16 07:18:45 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Reloading filter
Sep 16 07:18:45 FIREWALL rc.gateway_alarm[65772]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:5.947ms RTTsd:11.776ms Loss:21%)
Sep 16 07:18:45 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep 16 07:18:45 FIREWALL check_reload_status[635]: Reloading filter
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Sep 16 07:18:46 FIREWALL php-fpm[51827]: /rc.dyndns.update: phpDynDNS (@.mydomain.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 16 07:18:50 FIREWALL ppp[53627]: [wan_link0] LCP: no reply to 1 echo request(s)
Sep 16 07:19:00 FIREWALL ppp[53627]: [wan_link0] LCP: no reply to 2 echo request(s)
Sep 16 07:19:05 FIREWALL rc.gateway_alarm[23895]: >>> Gateway alarm: MNG_DHCP (Addr:192.168.2.1 Alarm:1 RTT:4.611ms RTTsd:15.937ms Loss:22%)

              Where all gateways start to indicate failures and the pppoe goes down. Effectively no traffic is passing from that point.

              But there are no lower level errors, the NICs do not show loss of link for example.

              The firewall is still logging and running scripts it doesn't appear to be down. At least until the end of that log.

              When did you try to connect? How did you connect?

              LaxarusL 1 Reply Last reply Reply Quote 0
              • LaxarusL
                Laxarus @stephenw10
                last edited by

                @stephenw10 I have further tweaked the ovpn gateway.

                Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Sep 16 07:18:46 FIREWALL php-fpm[20435]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.

                here, why is firewall trying to get routing for ipv6, the tunnel is ipv4 only.
                77982d0c-6409-4cad-9156-985e1a2d294c-image.png
                I had to power reset just to get everything back around 10:46.
                64a4a208-8a92-48ff-8f9e-736090de796f-system3.zip

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Just to be clear though you are unable to see any response from the firewall even on the local physical firewall?

                  It's extremely unusual to see it still logging and running scripts at that time but unresponsive at the console. Like I don't think I've ever seen that.

                  LaxarusL 1 Reply Last reply Reply Quote 0
                  • LaxarusL
                    Laxarus @stephenw10
                    last edited by

                    @stephenw10 said in Multiple issues, firewall freezes and whole network goes down.:

                    Like I don't think I've ever seen that

                    the attached capture for unresponsive console
                    9979fb96-2531-417e-a50a-dd6d321f8e90-pfsense freeze.zip

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Ok that looks like the IPMI console? And I assume that usually works as expected?

                      Is it configured for video as the primary console?

                      Are you able to test using a physical console?

                      LaxarusL 1 Reply Last reply Reply Quote 0
                      • LaxarusL
                        Laxarus @stephenw10
                        last edited by

                        @stephenw10 Yep, normally, it works without a problem. I am not sure how it is configured since there are no options to change the behavior.

                        No, I cannot access the physical console since it is at remote site.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          I mean if you check System > Advanced > Admin Access is the primary console set as video? Is serial even enabled?

                          LaxarusL 1 Reply Last reply Reply Quote 0
                          • LaxarusL
                            Laxarus @stephenw10
                            last edited by

                            @stephenw10 it is set as video console

                            01d6178c-5286-4860-9757-13a5ab64dcdb-image.png

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Hmm, very odd then.

                              I assume other parts of the IPMI is working at that time? It doesn't fail entirely?

                              LaxarusL 1 Reply Last reply Reply Quote 0
                              • LaxarusL
                                Laxarus @stephenw10
                                last edited by

                                @stephenw10 I had no issue with IPMI at all. Maybe clean install pfsense when I get back to the remote site again?

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  Worth trying.

                                  Any chance it's a network flood of some sort? Do you have any logging from attached switches?

                                  I could just about imagine that presenting like that given sufficient packet numbers.

                                  LaxarusL 1 Reply Last reply Reply Quote 0
                                  • LaxarusL
                                    Laxarus @stephenw10
                                    last edited by

                                    @stephenw10 I don't believe so. So far, I can only say that something is happening when pppoe goes down.
                                    I can try to disable bridge mode on the modem and DMZ the firewall which will enable me to take the pppoe out of the equation. However, my WAN IP will not be correct in that case.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      It looks more like the PPPoE fails because igb0 stops passing traffic to me. Mostly because at least one other NIC also stops passing traffic at that time. They are both igb NICs.

                                      A good test might be setup some other NIC type and test that at the time. Even better if it's a NIC on a different PCIe bus.

                                      LaxarusL 1 Reply Last reply Reply Quote 0
                                      • LaxarusL
                                        Laxarus @stephenw10
                                        last edited by

                                        @stephenw10 I can try ixl2 but 10g is overkill =:)

                                        1 Reply Last reply Reply Quote 0
                                        • LaxarusL
                                          Laxarus
                                          last edited by

                                          @stephenw10 It's been a while since this happened since my WAN was pretty stable. However, it happened again and most likely due to the weekly scheduled restart of the Modem. The weird thing is, for the previous scheduled restarts of the modem, this did not happen.

                                          The show starts at Oct 14 06:00. Until then, everything is normal. Full log attached.
                                          system.log.0
                                          Some extracts

                                          Oct 14 06:01:01 FIREWALL kernel: igb1: link state changed to DOWN
Oct 14 06:01:02 FIREWALL php-fpm[55721]: /rc.linkup: Hotplug event detected for MODEM(opt2) dynamic IP address (4: dhcp)
Oct 14 06:01:02 FIREWALL php-fpm[55721]: /rc.linkup: DEVD Ethernet detached event for opt2
Oct 14 06:01:03 FIREWALL rc.gateway_alarm[15550]: >>> Gateway alarm: MODEM_DHCP (Addr:192.168.0.1 Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:03 FIREWALL check_reload_status[635]: updating dyndns MODEM_DHCP
Oct 14 06:01:03 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:03 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Oct 14 06:01:03 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:01:05 FIREWALL dhcpleases[36990]: Could not deliver signal HUP to process 613: No such process.
Oct 14 06:01:05 FIREWALL check_reload_status[635]: Linkup starting igb1
Oct 14 06:01:05 FIREWALL kernel: igb1: link state changed to UP
Oct 14 06:01:05 FIREWALL rc.gateway_alarm[40725]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:05 FIREWALL rc.gateway_alarm[40886]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:05 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Oct 14 06:01:05 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:05 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Oct 14 06:01:05 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:01:05 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:07 FIREWALL php-fpm[89488]: /rc.dyndns.update: phpDynDNS (@.mydomain.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Oct 14 06:01:07 FIREWALL php-fpm[762]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:07 FIREWALL php-fpm[762]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MODEM_DHCP.
Oct 14 06:01:07 FIREWALL php-fpm[83954]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:07 FIREWALL php-fpm[83954]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Oct 14 06:01:09 FIREWALL rc.gateway_alarm[85414]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:09 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Oct 14 06:01:09 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:09 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Oct 14 06:01:09 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:01:10 FIREWALL php-fpm[762]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:10 FIREWALL php-fpm[762]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Oct 14 06:01:16 FIREWALL ppp[84675]: caught fatal signal TERM
Oct 14 06:01:16 FIREWALL ppp[84675]: [wan] IFACE: Close event
Oct 14 06:01:16 FIREWALL ppp[84675]: [wan] IPCP: Close event
Oct 14 06:01:16 FIREWALL ppp[84675]: [wan] IPCP: state change Opened --> Closing
Oct 14 06:01:16 FIREWALL ppp[84675]: [wan] IPCP: SendTerminateReq #4
Oct 14 06:01:16 FIREWALL ppp[84675]: [wan] IPCP: LayerDown
Oct 14 06:01:16 FIREWALL php-cgi[48940]: rc.kill_states: rc.kill_states: Removing states for IP {redactedIP}/32
Oct 14 06:01:18 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:01:18 FIREWALL php-fpm[95879]: /rc.linkup: Hotplug event detected for MODEM(opt2) dynamic IP address (4: dhcp)
Oct 14 06:01:18 FIREWALL php-fpm[95879]: /rc.linkup: DEVD Ethernet attached event for opt2
Oct 14 06:01:18 FIREWALL php-fpm[95879]: /rc.linkup: HOTPLUG: Configuring interface opt2
Oct 14 06:01:20 FIREWALL rc.gateway_alarm[15305]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:20 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Oct 14 06:01:20 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:20 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Oct 14 06:01:20 FIREWALL php-cgi[48940]: rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Oct 14 06:01:20 FIREWALL check_reload_status[635]: Rewriting resolv.conf
Oct 14 06:01:20 FIREWALL ppp[84675]: [wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Oct 14 06:01:20 FIREWALL ppp[84675]: [wan] IFACE: Down event
Oct 14 06:01:20 FIREWALL ppp[84675]: [wan] IFACE: Rename interface pppoe0 to pppoe0
Oct 14 06:01:20 FIREWALL ppp[84675]: [wan] IFACE: Set description "WAN"
Oct 14 06:01:21 FIREWALL ppp[84675]: [wan] IPCP: SendTerminateReq #5
Oct 14 06:01:21 FIREWALL ppp[84675]: [wan_link0] LCP: no reply to 1 echo request(s)
Oct 14 06:01:21 FIREWALL php-fpm[762]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:21 FIREWALL php-fpm[762]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Oct 14 06:01:22 FIREWALL ppp[84675]: [wan] Bundle: Shutdown
Oct 14 06:01:23 FIREWALL ppp[84675]: [wan_link0] Link: Shutdown
Oct 14 06:01:23 FIREWALL ppp[84675]: process 84675 terminated
Oct 14 06:01:24 FIREWALL rc.gateway_alarm[59440]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Oct 14 06:01:24 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Oct 14 06:01:24 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:24 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Oct 14 06:01:24 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:01:25 FIREWALL php-fpm[83954]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:25 FIREWALL php-fpm[83954]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Oct 14 06:01:29 FIREWALL check_reload_status[635]: Linkup starting igb1
Oct 14 06:01:29 FIREWALL kernel: igb1: link state changed to DOWN
Oct 14 06:01:32 FIREWALL check_reload_status[635]: Linkup starting igb1
Oct 14 06:01:33 FIREWALL kernel: igb1: link state changed to UP
Oct 14 06:01:53 FIREWALL check_reload_status[635]: rc.newwanip starting igb1
Oct 14 06:01:53 FIREWALL php-fpm[95879]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:53 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:01:55 FIREWALL php-fpm[55721]: /rc.newwanip: rc.newwanip: Info: starting on igb1.
Oct 14 06:01:55 FIREWALL php-fpm[55721]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.0.2) (interface: MODEM[opt2]) (real interface: igb1).
Oct 14 06:01:55 FIREWALL dhcpleases[26526]: Could not deliver signal HUP to process 13480: No such process.
Oct 14 06:01:55 FIREWALL dhcpleases[49055]: Could not deliver signal HUP to process 13480: No such process.
Oct 14 06:01:56 FIREWALL check_reload_status[635]: updating dyndns opt2
Oct 14 06:01:56 FIREWALL ppp[82148]: Multi-link PPP daemon for FreeBSD
Oct 14 06:01:56 FIREWALL ppp[82148]:  
Oct 14 06:01:56 FIREWALL ppp[82148]: process 82148 started, version 5.9
Oct 14 06:01:56 FIREWALL ppp[82148]: web: web is not running
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan] Bundle: Interface ng0 created
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan_link0] Link: OPEN event
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan_link0] LCP: Open event
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan_link0] LCP: state change Initial --> Starting
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan_link0] LCP: LayerStart
Oct 14 06:01:56 FIREWALL ppp[82148]: [wan_link0] PPPoE: Connecting to ''
Oct 14 06:01:56 FIREWALL kernel: ng0: changing name to 'pppoe0'
Oct 14 06:01:59 FIREWALL php-fpm[95879]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Oct 14 06:01:59 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Oct 14 06:02:00 FIREWALL dhcpleases[99884]: Could not deliver signal HUP to process 59176: No such process.
Oct 14 06:02:02 FIREWALL check_reload_status[635]: updating dyndns wan
Oct 14 06:02:02 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:02:02 FIREWALL check_reload_status[635]: Reloading filter
Oct 14 06:02:02 FIREWALL php-fpm[94330]: /rc.linkup: Hotplug event detected for MODEM(opt2) dynamic IP address (4: dhcp)
Oct 14 06:02:02 FIREWALL php-fpm[94330]: /rc.linkup: DEVD Ethernet detached event for opt2
Oct 14 06:02:03 FIREWALL ppp[82148]: PPPoE: rec'd ACNAME "TT-35-HATAY-ZTE-ZXR-02"
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] PPPoE: connection successful
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] Link: UP event
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: Up event
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: state change Starting --> Req-Sent
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: SendConfigReq #1
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   PROTOCOMP
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MRU 1492
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MAGICNUM 0x57649364
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: rec'd Configure Request #1 (Req-Sent)
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MAGICNUM 0x26b4014d
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MRU 1492
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   AUTHPROTO PAP
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: SendConfigAck #1
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MAGICNUM 0x26b4014d
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MRU 1492
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   AUTHPROTO PAP
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: state change Req-Sent --> Ack-Sent
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: rec'd Configure Reject #1 (Ack-Sent)
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   PROTOCOMP
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0] LCP: SendConfigReq #2
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MRU 1492
Oct 14 06:02:03 FIREWALL ppp[82148]: [wan_link0]   MAGICNUM 0x57649364

                                          This is becoming unbearable for me since I need this remote system to be extra stable. Any help at this point is very much appreciated.
                                          Do I need some professional help at this point? Can TAC lite help with this? I am not shy from spending some $$ to get this fixed at this point. I am even considering switching to opnsense but I like pfsense way better and unless as a last resort I will not.

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            So it was completely unresponsive again? You had to power cycle it?

                                            You are still using igb1 for WAN. Are you able to try a different NIC?

                                            It's a interesting log. The igb NIC bounces twice causing a bunch of service restarts but then remains stable.
                                            The PPP daemon then tries to reconnect but initially it fails with:

                                            Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0] PAP: rec'd NAK #1 len: 27
Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0]   MESG: Authentication failed!
Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0] LCP: authorization failed

                                            But then after ~40s:

                                            Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0] PAP: rec'd ACK #1 len: 39
Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0]   MESG: Authentication Successful,Welcome!
Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0] LCP: authorization successful

                                            That implies the NIC is passing traffic and the remote server is responding at that point. The PPPoE link comes up correctly.
                                            But only for ~90s, then:

                                            Oct 14 06:03:54 FIREWALL ppp[73450]: [wan_link0] LCP: no reply to 1 echo request(s)

                                            After that the ppp daemon times out and cannot reconnect again. But it keeps trying.

                                            Nothing is logged that seems be anything that would cause it to stop passing traffic.

                                            LaxarusL 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post