Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    metronet fiber, internet goes down roughly every 24 hours

    Scheduled Pinned Locked Moved General pfSense Questions
    45 Posts 5 Posters 9.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bcruze
      last edited by

      Time.txt

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        I don't see a 10.100.x.x IP address those there. you mean the 100.92.x.x address? That looks like CGN.

        It looks like the WAN address is changing, which is not unusual. The addresses involved there seem to be the WAN and it's gateway, I would not expect to see an IP for the local modem device at all.

        The DHCP logs are hard to read in that format. We are only interested in the dhclient entries. They might be in UTC vs whatever timezone your other logs are in. Really we need to see the dhclient entries in the dhcp log and the system log entries covering that same time span to see what triggered the renewal, or lack of it.

        If you're hitting that bug I expect to see the link fail or the lease force renewed for some reason in the system log.
        Then the dhclient start in the dhcp log but fail to get an IP and timeout.
        Then some error in the system log.
        Then nothing until you manually restart dhclient.

        I'm not seeing that though, from what we have so far it's more like the remote dhcp server is handing you a ludicrously long lease and then expecting it to renew sooner. This entry:

        dhclient
        93554
        bound to 100.92.204.194 -- renewal in 43200 seconds.
        

        Shows a 12h lease though. pfSense would normally attempt to renew it after half the lease time. If you filter the dhcp log for only dhclient process entries I expect to see it renewing every 6h.

        Steve

        1 Reply Last reply Reply Quote 0
        • B
          bcruze
          last edited by bcruze

          fixed the IP. this is carrier grade nat...

          is there a way i can get you the logs you need by the var/log area easier?

          Feb 23 09:35:29 pfSense dhclient[29691]: connection closed
          Feb 23 09:35:29 pfSense dhclient[29691]: exiting.
          Feb 23 09:35:31 pfSense dhclient: PREINIT
          Feb 23 09:35:32 pfSense dhclient[54752]: DHCPREQUEST on mvneta2 to 255.255.255.255 port 67
          Feb 23 09:35:32 pfSense dhclient[54752]: DHCPACK from 100.92.192.3
          Feb 23 09:35:32 pfSense dhclient: REBOOT
          Feb 23 09:35:32 pfSense dhclient: Starting add_new_address()
          Feb 23 09:35:32 pfSense dhclient: ifconfig mvneta2 inet 100.92.204.194 netmask 255.255.224.0 broadcast 100.92.223.255
          Feb 23 09:35:32 pfSense dhclient: New IP Address (mvneta2): 100.92.204.194
          Feb 23 09:35:32 pfSense dhclient: New Subnet Mask (mvneta2): 255.255.224.0
          Feb 23 09:35:32 pfSense dhclient: New Broadcast Address (mvneta2): 100.92.223.255
          Feb 23 09:35:32 pfSense dhclient: New Routers (mvneta2): 100.92.192.1
          Feb 23 09:35:32 pfSense dhclient: Adding new routes to interface: mvneta2
          Feb 23 09:35:32 pfSense dhclient: Creating resolv.conf
          Feb 23 09:35:32 pfSense dhclient[54752]: bound to 100.92.204.194 -- renewal in 43200

          another:
          Feb 23 14:29:03 pfSense dhclient[8995]: send_packet: No route to host
          Feb 23 14:29:30 pfSense dhclient[8995]: connection closed
          Feb 23 14:29:30 pfSense dhclient[8995]: exiting.
          Feb 23 09:29:39 pfSense dhclient: PREINIT
          Feb 23 09:29:39 pfSense dhclient[29287]: DHCPREQUEST on mvneta2 to 255.255.255.255 port 67
          Feb 23 09:29:39 pfSense dhclient[29287]: DHCPACK from 100.92.192.3
          Feb 23 09:29:39 pfSense dhclient: REBOOT
          Feb 23 09:29:39 pfSense dhclient: Starting add_new_address()
          Feb 23 09:29:39 pfSense dhclient: ifconfig mvneta2 inet 100.92.204.194 netmask 255.255.224.0 broadcast 100.92.223.255
          Feb 23 09:29:39 pfSense dhclient: New IP Address (mvneta2): 100.92.204.194
          Feb 23 09:29:39 pfSense dhclient: New Subnet Mask (mvneta2): 255.255.224.0
          Feb 23 09:29:39 pfSense dhclient: New Broadcast Address (mvneta2): 100.92.223.255
          Feb 23 09:29:39 pfSense dhclient: New Routers (mvneta2): 100.92.192.1
          Feb 23 09:29:39 pfSense dhclient: Adding new routes to interface: mvneta2
          Feb 23 09:29:39 pfSense dhclient: /sbin/route add default 100.92.192.1
          Feb 23 09:29:39 pfSense dhclient: Creating resolv.conf
          Feb 23 09:29:39 pfSense dhclient[29287]: bound to 100.92.204.194 -- renewal in 1800 seconds.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Is that two different devices? On two separate connections? They have the same IP address but only 6mins apart....

            One is given a 12h lease but the other only 30mins.

            I think you may have some conflict there....

            1 Reply Last reply Reply Quote 0
            • B
              bcruze
              last edited by

              same equipment. the new sg 3100.

              is there something i can change or look at to remedy this?

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                So that is two excerpts from the same dhcp log on the same device?

                The time stamps are confusing, what order should those be read in?

                Both those show successful renewal though which is not what is expected from the dhcp bug we referenced.

                Steve

                1 Reply Last reply Reply Quote 0
                • B
                  bcruze
                  last edited by bcruze

                  same 3100. same device

                  i pulled them directly from var\logs i copied them to a word file. ctrl + f what you wanted to see and pasted it here

                  read top to bottom

                  i am really hoping a static IP address from the provider will resolve this

                  called my isp to see if they could enable the static today. not possible i have to wait until tomorrow at 9am EST :(

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    In the dhcp logs view you can filter by the dhclient process and then just copy/paste them here directly without going through Word (or any other editor).
                    Some of those logs show the gateway not responding to ARP which probably won't be solved by using a static IP. If you can get one though if will obviously solve any dhcp issues.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • B
                      bcruze
                      last edited by

                      static ip has been set and active for over 24 hours now. NO issues whatsoever. first time ever with this new internet service.

                      i will look forward to when Internet Systems Consortium DHCP Server 4.3.6-P1 is updated to 4.4.2 within Pfsense per https://www.isc.org/dhcp/

                      so all of this is fixed for users.

                      so to continue using Pfsense i will be paying 10 dollars extra a month until the release..

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        It's dhclient not the dhcp server. It will be in 2.5 when that is released, it isn't yet in 2.5 snaps as they are currently built on 12.0.

                        Steve

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          As I said (in the wrong thread)....

                          stephenw10 Netgate Administrator about 16 hours ago

                          Ok, good news. The binary part of the fix for this is now in 2.4.5 snapshots:
                          https://github.com/pfsense/FreeBSD-src/commits/RELENG_2_4_5/sbin/dhclient/dhclient.c

                          The full fix also requires changes to the dhclient script which can be applied via the system patches package. I have briefly tested that and it didn't seem to break anything.

                          That patch is here: https://redmine.pfsense.org/attachments/download/2682/pfsense-dhclient-script-patch.txt

                          If you're able to test it we may be able to include it in 2.4.5.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • B
                            bcruze
                            last edited by

                            no problem Sir.

                            at this time i have no way to test as i am locked in a one year agreement with a static WAN ip address. my issue resolved.

                            not sure i provided you any good information. feel free to lock this thread and work with the other gentlemen if that seems best

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              I'm going to test locally but I can only try to simulate a failed dhcp server. It is definitely a bug that would be very good to squash. I'd love to hear from anyone who is hitting it 'in the field'.

                              Steve

                              A 1 Reply Last reply Reply Quote 0
                              • A
                                abovebrew @stephenw10
                                last edited by

                                @stephenw10 Hello, I feel like I'm running into this.

                                Quick overview on my situation:

                                • Currently running ver 22.05-RELEASE on a 2100. (recently updated)
                                • Ran over 2 years w/o ever having to reboot appliance on Xfinity internet.
                                • Recent switch to MetroNet Fiber - Internet goes down every 24-36 hours.
                                • Has happened 3 times so far, feels like it is reproducing reliably within these time frames.
                                • Manually releasing WAN / renew WAN action on WAN interface - Restores Internet

                                Trying to figure out a good path forward:

                                • Pay $10/month for a static IP (I'd rather not)
                                • Can we script something to automate the release/renew action during the night on a daily basis? (Feels like a hack)
                                • Is this a bug that can be fixed? (best solution IMO)
                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  What do you see logged when it happens?

                                  The actual bug referenced here was fixed in 2.4.5:
                                  https://redmine.pfsense.org/issues/9267

                                  I suggest you may well be hitting this:
                                  https://forum.netgate.com/post/1063443

                                  So specifically MetroNets broken DHCP relay behaviour. The workaround shown there should prevent it if so.

                                  Steve

                                  A 1 Reply Last reply Reply Quote 0
                                  • A
                                    abovebrew @stephenw10
                                    last edited by

                                    @stephenw10

                                    I haven't caught logs when the issue happens yet. I'm hoping I can catch on the next reproduce since I "sort of" know what to look for. Below is what I have on the most recent restore of services.

                                    Any logging in particular you would need?

                                    I feel like you're probably right about it being an issue with MetroNet. I called them today and they just rebooted everything and suggested I move to another piece of equipment that they can support.

                                    Appreciate you linking me to that HOWTO post. I'll probably end up implementing that after I reproduce one more time.

                                    This was on the latest restore:
                                    (Post was flagged for SPAM, so created a pastebin)
                                    https://pastebin.com/04r3wLek

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Yeah I bet it is that issue. The only way to be sure there is to packet capture the DHCP requests
                                      and check that unicast packets are being ignored but broadcasts see responses.

                                      M A 2 Replies Last reply Reply Quote 0
                                      • M
                                        michmoor LAYER 8 Rebel Alliance @stephenw10
                                        last edited by michmoor

                                        @stephenw10 The solution posted in that forum post - do you think a note can be added in the documentation?

                                        "Turn gateway monitoring back on. Your issue is not with that. It's with Metronet DHCP relays not responding to unicast renewals, the logs just confirm my suspicions. Perform the following. Goto interfaces > WAN and under DHCP client configuration check the box "Advanced configuration" and under presets select FreeBSD default. Then further down under Lease requirements and requests in the box "Option Modifiers" enter the following supersede dhcp-server-identifier 255.255.255.255

                                        Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                        Routing: Juniper, Arista, Cisco
                                        Switching: Juniper, Arista, Cisco
                                        Wireless: Unifi, Aruba IAP
                                        JNCIP,CCNP Enterprise

                                        1 Reply Last reply Reply Quote 1
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by stephenw10

                                          Yes, something needs to be added there. At the very least it needs more eyes to assess whether or not what they are doing can ever be valid. And if it is what we could/should be doing to address that. Let me open a ticket....

                                          Actually that dhclient option was added specifically to address this issue:
                                          https://redmine.pfsense.org/issues/7416

                                          Mmm, not sure where I'd expect a note in the docs to be. In the config override section here maybe?
                                          https://docs.netgate.com/pfsense/en/latest/interfaces/configure-ipv4.html#dhcp

                                          1 Reply Last reply Reply Quote 1
                                          • A
                                            abovebrew @stephenw10
                                            last edited by abovebrew

                                            @stephenw10

                                            Believe you are 100% correct here. I didn't reproduce my symptom, but feel like manually restarting the interface proves what's happening.

                                            In the packet captures, when source is my IP only DHCP Request packets get sent out with no DHCK ACK's (I saw these when I left packet capture going overnight). When source is 0.0.0.0 and destination is 255.255.255.255 I get DHCP ACK's back (saw this when manually restarting the WAN interface)

                                            Your help is much appreciated.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.