Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.2.4 to 2.3 dhcp no longer obtained until reboot

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    19 Posts 4 Posters 4.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Y
      yaboc
      last edited by

      hi i have strange issue and not sure if anyone experienced this after 2.2.x to 2.3 upgrade.
      after some amount of time my PCs hooked up to the switch lose network connectivity and they don't get address from pfsense when unplugged/plugged. when i reboot the router everything comes back online. setting static IP on a pc doesn't reach the pfsense box. i can reach it via ipsec from another pfsense and reboot it. i think this is happening on all three upgraded routers. any ideas? thanks

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        Sounds like the LAN's completely disconnecting from the internal network. Does everything on the LAN drop offline when you can no longer renew DHCP? Check Status>Interfaces, see if it has link. Diag>ARP, see if it has anything active on the LAN at the time it's not working.

        What hardware?

        1 Reply Last reply Reply Quote 0
        • Y
          yaboc
          last edited by

          hardware is
          Intel(R) Atom(TM) CPU D525 @ 1.80GHz
          4 CPUs: 1 package(s) x 2 core(s) x 2 HTT 2GB RAM SSD BOOT

          it happened again this morning and i was able to access router and i believe other devices through IPSEC
          LAN interface was up and ARP table was populated. It's very strange i'm thinking about doing clean 2.3 install and if it continues happening i'll go back to 2.2.6. Seems like a lot of people have weird things happening after 2.3 upgrade vs clean.

          1 Reply Last reply Reply Quote 0
          • C
            cmb
            last edited by

            The upgrade vs. clean issues are entirely in old package leftovers and those wouldn't have an impact like that.

            What NICs do you have in it?

            When it's not working, packet capture on the LAN, what traffic do you see?

            1 Reply Last reply Reply Quote 0
            • Y
              yaboc
              last edited by

              they're all

              em0:

              1 Reply Last reply Reply Quote 0
              • Y
                yaboc
                last edited by

                so it happened again

                arp table is empty except 10.18.66.1 which is the router and wan vips

                lan / wan interface is up

                packet capture on lan shows

                08:18:11.832770 ARP, Request who-has 10.18.66.8 tell 10.18.66.1, length 28
                08:18:12.221318 ARP, Request who-has 10.18.66.15 tell 10.18.66.1, length 28
                08:18:12.443232 ARP, Request who-has 10.18.66.10 tell 10.18.66.1, length 28
                08:18:12.951890 ARP, Request who-has 10.18.66.8 tell 10.18.66.1, length 28
                08:18:13.147215 ARP, Request who-has 10.18.66.6 tell 10.18.66.1, length 28
                08:18:13.355686 ARP, Request who-has 10.18.66.10 tell 10.18.66.1, length 28
                08:18:13.720475 ARP, Request who-has 10.18.66.15 tell 10.18.66.1, length 28
                08:18:14.004630 ARP, Request who-has 10.18.66.15 tell 10.18.66.1, length 28
                08:18:14.008446 ARP, Request who-has 10.18.66.8 tell 10.18.66.1, length 28
                08:18:14.157062 ARP, Request who-has 10.18.66.6 tell 10.18.66.1, length 28
                08:18:14.570169 ARP, Request who-has 10.18.66.10 tell 10.18.66.1, length 28
                08:18:15.113532 ARP, Request who-has 10.18.66.8 tell 10.18.66.1, length 28

                again im able to access the router through ipsec tunnel and reboot it which fixes it until the next time it happens.

                after the reboot arp is populated again and packet capture shows activity on LAN interface.

                also i believe this node was 2.2.4 to 2.3 and i didnt remove the only two packages before upgrade: pfBlockerNG and siproxd.

                1 Reply Last reply Reply Quote 0
                • Y
                  yaboc
                  last edited by

                  it keeps happening. i do have two dhcp pools on the subnet for pc/phones with deny in mac control (partial phone mac) on the main PC pool. but that works after reboot. i turned it off for testing. i guess nuking with backup is the last resort

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    That shows it's sending traffic and not receiving anything. Do you have kern.ipc.nmbclusters configured in /boot/loader.conf or loader.conf.local? If not, add this line to loader.conf.local:

                    kern.ipc.nmbclusters=1000000

                    and reboot. With e1000 NICs I doubt you're exhausting mbufs, but that's a potential symptom in that circumstance.

                    If it happens again with that, try running (where em1 is your LAN NIC, replace accordingly):

                    ifconfig em1 down
ifconfig em1 up

                    and see if that does anything. That may narrow it down further.

                    1 Reply Last reply Reply Quote 0
                    • Y
                      yaboc
                      last edited by

                      kern.ipc.nmbclusters=10000
                      was set in loader.conf.local, i changed the value to 1000000 per your post and the system didn't come back online. no ssh/ipsec/wan response. i have to go on site and change it back to 10000.

                      1 Reply Last reply Reply Quote 0
                      • Y
                        yaboc
                        last edited by

                        after setting kern.ipc.nmbclusters=1000000 system never booted up it was hanging at
                        run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_action

                        i tried to set the variable back to 10000 preboot and it was still doing the same. decided to intall 2.3.0 fresh and restore the config. so far so good. i'll repost if the issue continues.

                        1 Reply Last reply Reply Quote 0
                        • Y
                          yaboc
                          last edited by

                          fresh install of 2.3 and restore the config with only pfblocker and siprox still does the same thing. LAN loses internet and reboot helps until the next time it happens. i'm going to switch back to 2.2.4 which was the rev i started this upgrade with.

                          1 Reply Last reply Reply Quote 0
                          • X
                            xtofh
                            last edited by

                            Same problem here. After a few hours, on multiple devices, all with same nic+cpu.

                            08:57:29.526111 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:b5:6d:00:d8:09, length 300
08:57:34.383325 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:b5:6d:00:d8:09, length 300
08:57:39.394949 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:b5:6d:00:d8:09, length 300

                            Same driver.

                            em1: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k=""> port 0x2000-0x201f mem 0xf7c00000-0xf7c1ffff,0xf7c20000-0xf7c23fff irq 18 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: f4:90:ea:10:12:41
em1: netmap queues/slots: TX 1/1024, RX 1/1024</intel(r)>

                            System:

                            CPU: AMD GX-416RA SOC                                (1597.14-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x700f01  Family=0x16  Model=0x0  Stepping=1

                            [2.3-RELEASE][root@pf-cs]/root: cat /boot/loader.conf
autoboot_delay="3"
vm.kmem_size="536870912"
vm.kmem_size_max="1073741824"
kern.ipc.nmbclusters="512000"
vfs.zfs.prefetch_disable="1"
boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"
comconsole_speed="115200"
hw.usb.no_pf="1"
[2.3-RELEASE][root@pf-cs]/root: cat /boot/loader.conf.local
autoboot_delay="3"
vm.kmem_size="536870912"
vm.kmem_size_max="1073741824"
kern.ipc.nmbclusters="512000"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole"
hw.usb.no_pf="1"
vfs.zfs.prefetch_disable="1"
                            1 Reply Last reply Reply Quote 0
                            • X
                              xtofh
                              last edited by

                              Oh, and ifconfig em1 down and then up does help!

                              If you need anything else (tcpdumps, other stuff, etc..) let me know.

                              1 Reply Last reply Reply Quote 0
                              • X
                                xtofh
                                last edited by

                                Hi all, back with another update after a BIOS upgrade and a complete fresh install, but the problem remains.

                                I didn't see it yesterday but there are errors in the dhcp log:

                                Apr 22 08:53:50	dhcpd		dhcp.c:3763: Failed to send 300 byte long packet over em1 interface.
Apr 22 08:53:50	dhcpd		send_packet: No buffer space available
Apr 22 08:53:50	dhcpd		DHCPOFFER on 192.168.10.100 to 00:b5:6d:00:d8:09 (R9Z0F0V) via em1
Apr 22 08:53:50	dhcpd		DHCPDISCOVER from 00:b5:6d:00:d8:09 (R9Z0F0V) via em1

                                That does not seem good! Any suggestions on this?

                                Regards.

                                1 Reply Last reply Reply Quote 0
                                • C
                                  cmb
                                  last edited by

                                  Could you download the status tgz file from status.php and get it to me? Email cmb at pfsense dot org or PM me here to arrange transfer.

                                  1 Reply Last reply Reply Quote 0
                                  • X
                                    xtofh
                                    last edited by

                                    Thanks, emailed it  ;)

                                    1 Reply Last reply Reply Quote 0
                                    • H
                                      hdokes
                                      last edited by

                                      I too am having this same issue on every box I have upgraded to 2.3.  I have to say I have been pulling my hair out thinking I was the only one till I came across this thread.  I am using several different platforms but for the sake of uniformity and just trying to get to the heart of the problem, let's cover the machines I am using most.

                                      Firebox X750e and Firebox X1250e
                                      2gb RAM
                                      4GB CF
                                      pfSense 2.3 nano embedded
                                      LAN plugged into port 1 (sk0) of internal 4 ports
                                      WAN plugged into port 1 (msk0) of 4 port expansion card
                                      Firmware is upgraded and patched for serial port console during/after boot.

                                      Symptom:  The LAN connection goes out to lunch arbitrarily.  I can unplug the cable and plug it back in and the LAN port is active again.  Ultimately, it will go out to lunch again.  The connection and activity lights will go out typically when this is happening but not all the time.  When plugging the cable back in the lights come back every time if they were off.

                                      I figured it was an issue with plugins however I have reluctantly uninstalled all plugins however the issue persists.

                                      We were experiencing this issue with the WAN port as well but we did the disable of the hw.msk.msi_disable=1 and the WAN ports have not dropped out since.  Granted, it's just been a few days so far… knock on wood.

                                      I am going to move the LAN port over to an extra expansion port and see if there is a difference there.  Will report back.

                                      Let me know if there is any additional information that may assist in this troubleshooting process.

                                      Thanks

                                      1 Reply Last reply Reply Quote 0
                                      • Y
                                        yaboc
                                        last edited by

                                        we switched back to 2.2.x but it seems that

                                        https://forum.pfsense.org/index.php?topic=110710.30

                                        could be also a reason but i haven't' tried the workaround which is to only run it on 1 core. i'll wait for out of the box solution since these are production systems.

                                        Thanks

                                        1 Reply Last reply Reply Quote 0
                                        • H
                                          hdokes
                                          last edited by

                                          So after moving the LAN port over to the expansion card of the firebox we have not had a drop out on the LAN yet.  In addition, since doing the hw.msk.msi_disable=1 the WAN port still has not dropped out.  Any reason we should suspect the on board 4 port ethernet interface drivers as the culprit here?

                                          I'm also going to upgrade an existing vmware ESXi 5.1 box where I have pfsense running as a virtual firewall and see what happens there.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.