Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    WAN interfaces fail to return after power outage

    Scheduled Pinned Locked Moved
    General pfSense Questions
    4
    49
    6.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      peter_richardson
      last edited by peter_richardson

      Hi all,

      I have 2 WAN interfaces:

      igb0: WAN (DHCP) fixed wireless NBN connection
      igb1: WWAN (DHCP) 4G cellular connection via ethernet from a Netgear Nighthawk M1 MR1100

      When the power goes off I have noticed that both WAN connections remain "pending". They have received an IP address but there's no action. I checked the System Logs and under the General tab I find this:

      Sep 29 12:19:00
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:05
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:05
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:06
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:10
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:11
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:12
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

Sep 29 12:19:15
kernel
arpresolve: can't allocate llinfo for 100.65.48.1 on igb0

      etc etc

      If I disable one of the WAN interfaces and then enable it, the interface comes back online. So whenever the power goes out, I need to remember to log into pfSense, disable both WAN interfaces, re-enable them, and then we're back online.

      There doesn't seem to be any info in the System Log about igb1 WWAN.

      Running 2.4.3-RELEASE-p1 (amd64)

      Any ideas what's going on here?

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Is that the correct gateway address for your WAN connection?

        If not you might be getting a lease from the local modem until it detects the link is back up. In which case you can set DHCP to refuse leases from that device.

        Steve

        1 Reply Last reply Reply Quote 0
        • P
          peter_richardson
          last edited by

          Thanks Stephenw10 yes that's the correct gateway address for our WAN, it gets handed to us via DHCP from the ISP. There is no modem involved.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Hmm, well it appears it's not responding to ARP requests for some reason.

            Are you spoofing your MAC address maybe?

            When it's in that state try running ifconfig -v igb0

            And if possible run a packet capture ion that interface to see what it's actually sending and what's coming back (if anything).

            Steve

            1 Reply Last reply Reply Quote 0
            • P
              peter_richardson
              last edited by

              Thanks Steve.

              I'm not spoofing my MAC address.

              What exactly is it that's not responding to ARP requests?

              I'll do some testing and try to replicate it so I can do a capture. Thanks again.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                It's whatever is upstream. If you connect directly and pull a DHCP lease it's whatever that is giving you as a gateway, 100.65.48.1.

                Steve

                B 1 Reply Last reply Reply Quote 0
                • P
                  peter_richardson
                  last edited by peter_richardson

                  Seems that way, or might it be a problem with pfsense because it was happening with both WANs, 2 completely different service? And if it were a problem upstream, it wouldn't go away from just unplugging and replugging the cable, right?

                  1 Reply Last reply Reply Quote 1
                  • B
                    bfeitell @stephenw10
                    last edited by bfeitell

                    @stephenw10

                    Please see my post about a similar issue, and another thread I responded to. I think I know what is happening here, and it needs to be documented, and or added as a default setting in new installs.

                    It seems that the dhclient code has been changed to respect the "interface-mtu" option that is being issued via DHCP by some CMTS equipment.

                    I the cases I have seen the MTU is being set to 576, rather than being left at the pfSense default of 1500.

                    In addition, the interface-mtu option issued via DHCP takes precedence over an MTU explicitly set by the user. To override the bad interface-mtu being set via DHCP, dhclient must be instructed to ignore the option. This is done by setting supersede interface-mtu 0 in the "Option modifiers" section of "Lease Requirements and Requests".

                    When the MTU is set too small, it seems that DHCP renewals are failing. The failures coincide with errors like:
                    arpresolve: can't allocate llinfo for $GATEWAY on $INTERFACE

                    I think this is biting a lot of users on the upgrade to 2.4.4-RELEASE, and the symptoms are quirky. If left alone, the interface may stay up, but certain websites will become inaccessible due to packet fragmentation.

                    PRIOR POSTS ON SAME SUBJECT:
                    https://forum.netgate.com/topic/136089/solved-and-revised-2-4-4-release-arpresolve-can-t-allocate-llinfo-for-gateway-on-interface0-dhcp-mtu-576

                    https://forum.netgate.com/topic/136253/frequent-internet-loss-need-help-figuring-out-where-and-why-maybe-pfsense-modem-isp-or-all-3

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by stephenw10

                      Wow, nice catch. And what are they doing sending 576?! 😕

                      That does seem like it could be related at least. I'll be watching with anticipation...

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Actually that should have been resolved already in https://redmine.pfsense.org/issues/8507

                        If it does work that has somehow missed your install.

                        Steve

                        B 1 Reply Last reply Reply Quote 0
                        • B
                          bfeitell
                          last edited by bfeitell

                          From what I have read, I see references to the 576 MTU related to dialup connections. This might be an old fall-back that is being exposed only because dhclient now respects the option interface-mtu value being sent by the DHCP server. The value shows up in the issued lease. The changes in dhclient upstream are now exposing this.

                          This is worth exploring in connection with reports of WAN interface disconnections, unpredictable website connectivity, and may affect things like name resolution. When combined with the "IP Do-Not-Fragment compatibility" option in System/Advanced/Firewall&NAT, the small MTU breaks connectivity with some websites. I saw problems with the iHeart Radio website and streams, and with loading newyorker.com. Please propagate this up the chain. My earlier post has links to the issues as they are discussed in the FreeBSD development system.

                          https://forum.netgate.com/topic/136089/solved-and-revised-2-4-4-release-arpresolve-can-t-allocate-llinfo-for-gateway-on-interface0-dhcp-mtu-576

                          1 Reply Last reply Reply Quote 0
                          • B
                            bfeitell @stephenw10
                            last edited by bfeitell

                            @stephenw10

                            The fix discussed in Redmine doesn't seem to have made it into 2.4.4-RELEASE.

                            Rather, the fix of using the patched version of dhclient now in the FreeBSD tree is that the user must issue "supersede interface-mtu 0" to ignore the requested option 26 information. Dhclient is still requesting option 26 info from the DHCP server. The patch allows being able to supersede option 26 as issued with the lease.

                            1 Reply Last reply Reply Quote 0
                            • B
                              bfeitell
                              last edited by

                              I have opened a Redmine account, and posted in the relevant thread.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                I agree, it's certainly worth exploring. It could explain a number of threads here.

                                The value supersede interface-mtu 0 should be in the dhclient conf files in /var/etc by default. It is on everything I've just checked. If some connections are still seeing a 576 MTU then there must be some combination of factors that prevent it being added. If that is the case we need to find out what they are and stop that happening.

                                Steve

                                1 Reply Last reply Reply Quote 0
                                • B
                                  bfeitell
                                  last edited by

                                  I have a sneaking suspicion that a prior manual setting of MTU on the interface may be interfering with the the setting of supersede interface-mtu 0 in dhclient.conf on upgrade. I know that I have previously hard set the MTU to 1500 on a number of boxes as a matter of course. In this instance, the hard set MTU will not be respected if supersede interface-mtu 0 is not making it into dhclient.conf.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    I think you're right. Working on something now....

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      bfeitell
                                      last edited by

                                      Jim Pingle, the developer working on this has entered a new diff. Apparently, checking the advanced options checkbox and then saving and applying the config with no other changes entered, and then upgrading to 2.4.4-RELEASE, is enough to disrupt the fix the developers had put in place for the option 26 interface-mtu bug introduced by the new dhclient.

                                      1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by

                                        I added a new note with a workaround to the Upgrade Guide: https://www.netgate.com/docs/pfsense/install/upgrade-guide.html#upgrading-from-versions-older-than-pfsense-2-4-4

                                        A patch is available that can be added with the System Patches package.

                                        The fix is discussed on https://redmine.pfsense.org/issues/8507

                                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 1
                                        • B
                                          bfeitell
                                          last edited by

                                          Thank you! This is wonderful.

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            bfeitell
                                            last edited by bfeitell

                                            I think this bug also applies to fresh installs using a restored config, not just on in-place upgrades. That is the case for the system I encountered this on.

                                            jimpJ 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post