Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    WAN link going down

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    62 Posts 9 Posters 15.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Ok, so you could have something handing out address in that subnet. The 100.1 address is totally spurious, not used as a tunnel address etc?

      Steve

      1 Reply Last reply Reply Quote 0
      • T
        TieT
        last edited by

        Nope not used.

        But the WAN link seems stable for now.
        I made some changes in the System general settings and they seem to work.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          What did you change?

          Steve

          1 Reply Last reply Reply Quote 0
          • Q
            q54e3w
            last edited by

            I have the same issue and posted some logs in another thread yesterday. I've seen a few posts like this recently do possibly something needs tweaking. I'll add a link to my logs tomorrow when I get back to my desk. I've tried setting static address, disabling monitoring but every 3-7 days my line drops and needs s reboot of wan or box to resolve.

            Edit: here's the link to my logs… https://forum.pfsense.org/index.php?topic=88236.msg491251#msg491251

            1 Reply Last reply Reply Quote 0
            • T
              TieT
              last edited by

              In System -> general settings i added 127.0.0.1 as first dns server without any gateway
              then on the second line:
              8.8.8.8 with the gateway from my provider 84.192.192.1
              on the third:
              8.8.4.4 with 84.192.192.1

              the option to automatically find the gateway seems to f*ck the wan link sometimes…

              I didn't make any changes otherwise to the network config.

              Only updated LCDProc and used your guide to set it up because it kept crashing badly.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Interesting, thanks.

                1 Reply Last reply Reply Quote 0
                • X
                  xtofh
                  last edited by

                  Hi,

                  Are you sure this is resolved? I find it strange that changing dns settings solves this.

                  Looking at your IP gateway, we're using the same internet provider and I'm having the same issue. (only updated yesterday, noticed the link going down today)

                  I had a similar issue a few years back: https://forum.pfsense.org/index.php?topic=51420.0

                  It was solved by using the 2.1 release. But now the problem returned after updating to 2.2.

                  I'll keep an eye on it (it only happened once now) and will collect logs in case it returns.

                  Regards,
                  Kristof.

                  1 Reply Last reply Reply Quote 0
                  • X
                    xtofh
                    last edited by

                    Okay, just had this happen again, here's the logs from that time.

                    All VPNs going down..

                    Feb 25 13:53:01 pfsense kernel: ovpns23: link state changed to DOWN
Feb 25 13:53:01 pfsense php-fpm[89891]: /rc.newwanip: rc.newwanip: Info: starting on ovpns19.
Feb 25 13:53:01 pfsense php-fpm[89891]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.250.17) (interface: []) (real interface: ovpns19).
Feb 25 13:53:01 pfsense php-fpm[89891]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection -  ->  192.168.250.17 - Restarting packages.
Feb 25 13:53:01 pfsense check_reload_status: Starting packages
Feb 25 13:53:02 pfsense php-fpm[21603]: /rc.openvpn: OpenVPN: Resync server24 [CUST26] S2S
Feb 25 13:53:02 pfsense kernel: ovpns23: link state changed to UP
Feb 25 13:53:02 pfsense check_reload_status: rc.newwanip starting ovpns23
Feb 25 13:53:02 pfsense kernel: ovpns24: link state changed to DOWN
Feb 25 13:53:02 pfsense check_reload_status: Reloading filter

                    Our WAN interface.. (em1)

                    Feb 25 13:53:02 pfsense kernel: em1: Watchdog timeout -- resetting
Feb 25 13:53:02 pfsense kernel: em1: Queue(0) tdh = 801, hw tdt = 770
Feb 25 13:53:02 pfsense kernel: em1: TX(0) desc avail = 31,Next TX to Clean = 801

                    The rest of the VPNs going down. (we've got 30 or so)

                    Feb 25 13:53:02 pfsense php-fpm[21603]: /rc.openvpn: OpenVPN: Resync server25 [CUST25] S2S
Feb 25 13:53:02 pfsense kernel: ovpns24: link state changed to UP
Feb 25 13:53:02 pfsense check_reload_status: rc.newwanip starting ovpns24
Feb 25 13:53:02 pfsense kernel: ovpns25: link state changed to DOWN
Feb 25 13:53:02 pfsense kernel: em1: link state changed to DOWN
Feb 25 13:53:02 pfsense check_reload_status: Linkup starting em1
Feb 25 13:53:02 pfsense php-fpm[21603]: /rc.openvpn: OpenVPN: Resync server26 [CUST26] S2S
Feb 25 13:53:02 pfsense kernel: ovpns25: link state changed to UP
Feb 25 13:53:02 pfsense check_reload_status: rc.newwanip starting ovpns25
Feb 25 13:53:02 pfsense kernel: ovpns26: link state changed to DOWN
Feb 25 13:53:03 pfsense php-fpm[21603]: /rc.linkup: DEVD Ethernet detached event for wan
Feb 25 13:53:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 25 13:53:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 25 13:53:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 25 13:53:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1

                    And then it continues with the can't allocate llinfo.

                    I tried disconnecting and re-connecting the network cable: no result.
                    I tried rebooting the cable modem: no result.

                    It did try getting a new DHCP address:

                    
$ tcpdump -i em1 -n
14:04:25.808903 IP 81.82.209.44.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:27.003202 IP 81.82.209.44.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:31.003925 IP 81.82.209.44.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:40.009358 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:41.009469 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:42.000206 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300
14:04:44.001156 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0d:88:cc:fa:2f, length 300

                    But that's it, never got any.

                    Final step we took (because it was taking us +5 minutes already) was reboot the pfsense. After the reboot connection was good..

                    I will probably take a 2.1.5 to temporarily switch this one out (luckily it's in our building) but will leave the 2.2 in place so I can switch easily.

                    So I there's anything I can/should try, let me know, no problem.

                    Regards,
                    Kristof

                    1 Reply Last reply Reply Quote 0
                    • Q
                      q54e3w
                      last edited by

                      Id love to share with you a solution but Im seeing exactly the same thing here and can confirm its not fixed for me by manipulating DNS as TieT suggested fixed his. My issue is triggered by my ISP WAN dhcp address expiring every seven days. Rebooting the firewall is the easiest way to fix it, I spent a couple of hours last time it went down capturing packets and what not and it appears in my situation to be related to DHCP queries not syncing. It could be specific to my ISP and I'll keep capturing and investigating if/when it continues. Thanks for sharing your logs.

                      1 Reply Last reply Reply Quote 0
                      • X
                        xtofh
                        last edited by

                        I've put my 2.1.5 back in place. let's see how this goes. (I'm pretty sure this will stay up)

                        I also suspect it has something to do with the dhcp renewal.

                        2.1 solved it in 2012 for me :-) - https://forum.pfsense.org/index.php?topic=51420.0

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by 

                          Feb 25 13:53:02 pfsense kernel: em1: Watchdog timeout -- resetting

                          That is a pretty bad sign.

                          You might try looking at sysctl dev.em.1
                          There may be some error couters there that show something nasty happened.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • X
                            xtofh
                            last edited by

                            Ok, put a switch in between modem<->pfsense , recorded the current state of sysctl dev.em.1 (attached)

                            Currently it's good  :o (lol)  but if the problem returns I'll redo the sysctl dev.em.1 and won't reboot. (I'll just use a 2.1.5 device so can grab any logs needed of the 2.2)

                            Thanks, I'll keep you posted.

                            Kristof

                            devem1.txt

                            1 Reply Last reply Reply Quote 0
                            • J
                              JWTech
                              last edited by

                              I'm getting the same issue.

                              Seems my external IPv4, which is set by DHCP, is randomly disconnecting while the IPv6 address is fine.

                              No clue what's the cause, but I'm running the latest version on a Nano USB.

                              1 Reply Last reply Reply Quote 0
                              • X
                                xtofh
                                last edited by

                                Ok, not resolved yet :(

                                My current configuration is Modem <-> Switch <-> pfsense.

                                I did see some bad things in the sysctl.em.1 :

                                dev.em.1.mac_stats.collision_count: 0
dev.em.1.mac_stats.symbol_errors: 4294967295
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.missed_packets: 28403

                                My log was already full (I will make it bigger or use remote syslog) but I did see this:

                                .. first a ton of these messages:
Feb 28 11:21:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:04 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:05 pfsense kernel: em1: Watchdog timeout -- resetting
Feb 28 11:21:05 pfsense kernel: em1: Queue(0) tdh = 0, hw tdt = 993
Feb 28 11:21:05 pfsense kernel: em1: TX(0) desc avail = 31,Next TX to Clean = 0
Feb 28 11:21:05 pfsense kernel: em1: link state changed to DOWN
Feb 28 11:21:05 pfsense check_reload_status: Linkup starting em1
Feb 28 11:21:06 pfsense php-fpm[43360]: /rc.linkup: DEVD Ethernet detached event for wan
Feb 28 11:21:06 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:06 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:09 pfsense check_reload_status: Linkup starting em1
Feb 28 11:21:09 pfsense kernel: em1: link state changed to UP
Feb 28 11:21:10 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:10 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:11 pfsense php-fpm[43360]: /rc.linkup: Shutting down Router Advertisment daemon cleanly
Feb 28 11:21:11 pfsense php-fpm[54180]: /rc.linkup: DEVD Ethernet attached event for wan
Feb 28 11:21:11 pfsense php-fpm[54180]: /rc.linkup: HOTPLUG: Configuring interface wan
Feb 28 11:21:13 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:13 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
Feb 28 11:21:14 pfsense kernel: arpresolve: can't allocate llinfo for 81.82.192.1 on em1
.. and then more of these

                                This only happens on 2.2, the same configuration with 2.1.5, no problem. It took about 30 hours before it occured the first time.

                                Anything else I might try to resolve/further pinpoint this?

                                Regards,
                                Kristof.

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  New hardware?

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • X
                                    xtofh
                                    last edited by

                                    @Derelict:

                                    New hardware?

                                    No, only diiference: 2.1.5 -> 2.2.

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      @man:

                                      arpresolve: can't allocate llinfo for %d.%d.%d.%d The route for the ref-
                                          erenced host points to a device upon which ARP is required, but ARP was
                                          unable to allocate a routing table entry in which to store the host's MAC
                                          address.  This usually points to a misconfigured routing table.  It can
                                          also occur if the kernel cannot allocate memory.

                                      What's in netstat -rn when it's failing?

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        JWTech
                                        last edited by

                                        Just resolved my issue by disabling the local DNS resolution. Changed it to just use Google's DNS servers instead (8.8.8.8, 8.8.4.4) and it's working fine.

                                        Not sure if it's the exact same issue as what everyone else has had, but that's what did it for me.

                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          TieT
                                          last edited by

                                          Test it for a few days, it worked for me for a few days, but then the gateway was lost again.
                                          It seems when I create a vpn to my fw the issue begins…

                                          @JWTech:

                                          Just resolved my issue by disabling the local DNS resolution. Changed it to just use Google's DNS servers instead (8.8.8.8, 8.8.4.4) and it's working fine.

                                          Not sure if it's the exact same issue as what everyone else has had, but that's what did it for me.

                                          1 Reply Last reply Reply Quote 0
                                          • X
                                            xtofh
                                            last edited by

                                            @Derelict:

                                            @man:

                                            arpresolve: can't allocate llinfo for %d.%d.%d.%d The route for the ref-
                                                erenced host points to a device upon which ARP is required, but ARP was
                                                unable to allocate a routing table entry in which to store the host's MAC
                                                address.  This usually points to a misconfigured routing table.  It can
                                                also occur if the kernel cannot allocate memory.

                                            What's in netstat -rn when it's failing?

                                            Tnx for the suggestion, will try this and the sysctl dev.em.1 on next problem.

                                            I'm waiting for it to occur again (it can take several hours but it might as well take 48 hours) and report back.

                                            Kristof

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.