DHCP fails silently, but works on reboot of pfSense



  • I have an issue that has persisted between several installs of pfSense, CE on a VM, CE on bare hardware, and even the image for the XG-2758 from Netgate.

    Network:
    Two physical interfaces on the pfSense machine; WAN and LAN. On the LAN interface there are 200 VLANs defined, which we use to segment our network and allow a different subnet for each of the endpoints for the VLANs. Hence we can create firewall rules for each "VLAN interface" in pfSense.

    My problem is that ANY changes that modifies an interface results in a silent crash of the DHCP server, which then cannot be restarted (whether it's the LAN interface, a VPN interface or something else).
    Nothing is present in the logs, other than the startup of the DHCP server on each interface, and maybe a DHCPDISCOVER or two until it's restarted by the service watchdog package.
    Restarting the DHCP service does not work, however a restart of pfSense sometimes works (but not always!). We have however identified a procedure that allows DHCP to start on every reboot of pfSense:

    1. Reboot pfSense
    2. Disconnect interfaces from our backbone switch (there are 10 of these interfaces, each with 20 VLANS on it)
    3. Wait for pfSense to boot up, and DHCP service to start.
    4. Reconnect interfaces with a few seconds interval between connecting each interface.

    This procedure leads me to believe there's a problem with the service more or less being DDOS'ed by our hosts, but I'm not sure to verify if that's what causes the DHCP service to not want to restart.

    Anyone know what I can do here? It's quite annoying, as nothing can be done on the machine itself that modifies an interface in any way (such as creating a VPN server or even restarting one).



  • I have found a workaround so far by using the DHCP relay to an external DHCP server. Still very puzzled why the local DHCP daemon crashes on pfSense.



  • I'm seeing a very similar issue. DHCP just stops working... reboot come back online. restarting DHCP service alone does not do the trick. Running XG-1541 on 2.4.4


  • Rebel Alliance Developer Netgate

    There are no errors at all in the system log or DHCP log?

    The service shows as stopped under Status > Services?

    If you restart the service there, it doesn't work? (Or if it's running, stop and then start it again)

    Anything else unusual when it's down, like RAM usage?

    Are you using any other new/recent features like DNS over TLS?



  • @jimp

    1. The system log is littered with:
      Nov 6 08:10:55 check_reload_status updating dyndns optxx
      Nov 6 08:10:56 kernel vlanxx: changing name to 'igb4.xxxx'
      Nov 6 08:10:56 check_reload_status Restarting ipsec tunnels
    2. Big Red X next to DHCP Services
    3. Hitting the little blue "play" button gets the green gear spinning but then settles back at a red X with blue play button
    4. Memory usage is at 2%
    5. No DNS over TLS (using default Resolver

    I'm confused about the ipsec tunnels being constantly restarted as I have not started any VPN services. I am running 100+ VLANs.


  • Rebel Alliance Developer Netgate

    It's sounding like there is some kind of interface event happening that triggers it. Something is causing the link to bounce on that interface. Are you sure those are the only three messages that happen? Maybe if you show more log lines you'd see the actual culprit.

    Are any of the VLANs (or the parent interface) set to obtain their own IP address from DHCP, like a cable WAN? If so you might be hitting https://redmine.pfsense.org/issues/8507 if the interface has advanced DHCP client options enabled.



  • @jimp
    Every VLAN interface is setup for static IP configuration
    I am not sure why the "updating dyndns optxx" either unless apart of default DNS Resolver
    Also, the "kernel vlanxx" are vlan numbers I have not setup

    Nov 6 08:22:12 check_reload_status Restarting ipsec tunnels
    Nov 6 08:22:25 check_reload_status updating dyndns optxx
    Nov 6 08:22:25 kernel vlanxx: changing name to 'igb2.xxxx'
    Nov 6 08:22:25 check_reload_status Restarting ipsec tunnels
    Nov 6 08:22:37 check_reload_status updating dyndns optxx
    Nov 6 08:22:37 php-fpm 27426 /interfaces.php: Gateway, none 'available' for inet6, use the first one configured. ''
    Nov 6 08:22:37 kernel vlanxx: changing name to 'igb4.xxxx'
    Nov 6 08:22:37 check_reload_status Restarting ipsec tunnels
    Nov 6 08:22:49 check_reload_status updating dyndns optxx
    Nov 6 08:22:49 kernel vlanxx: changing name to 'igb4.xxxx'
    Nov 6 08:22:49 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:01 check_reload_status updating dyndns optxx
    Nov 6 08:23:02 kernel vlanxx: changing name to 'igb2.xxxx'
    Nov 6 08:23:02 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:14 check_reload_status updating dyndns optxx
    Nov 6 08:23:14 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:14 kernel vlanxx: changing name to 'igb2.xxxx'
    Nov 6 08:23:26 check_reload_status updating dyndns optxx
    Nov 6 08:23:27 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:27 kernel vlanxx: changing name to 'igb2.xxxx'
    Nov 6 08:23:39 check_reload_status updating dyndns optxx
    Nov 6 08:23:40 kernel vlanxx: changing name to 'igb2.xxxx'
    Nov 6 08:23:40 php-fpm 9855 /interfaces.php: Gateway, none 'available' for inet6, use the first one configured. ''
    Nov 6 08:23:40 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:52 check_reload_status updating dyndns optxx
    Nov 6 08:23:52 check_reload_status Restarting ipsec tunnels
    Nov 6 08:23:52 kernel vlanxx: changing name to 'igb2.xxxx'


  • Rebel Alliance Developer Netgate

    There must be something farther back in the logs that triggers all of that, though. Those are all the consequence of some kind of interface event (link up/down) or similar.



  • This is WAY off topic, so I apologize, and maybe it can be a separate topic.

    @obelsen - I'm very new to the use and tech, but why would there be a real-world use for so many VLAN's - you state 200 in your original post. How/when would anybody need to use that many virtual networks? I could see a handful being useful, but why a very large amount? I'm just curious...

    Jeff



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    There are no errors at all in the system log or DHCP log?

    The service shows as stopped under Status > Services?

    If you restart the service there, it doesn't work? (Or if it's running, stop and then start it again)

    Anything else unusual when it's down, like RAM usage?

    There are no errors in either the system log or the DHCP log.
    It is not possible to restart the service, only by restarting the machine. The service is stopped, and is not running. RAM usage is far from a problem on our XG-2758.

    Are you using any other new/recent features like DNS over TLS?

    No.

    @jimp said in DHCP fails silently, but works on reboot of pfSense:

    There must be something farther back in the logs that triggers all of that, though. Those are all the consequence of some kind of interface event (link up/down) or similar.

    While this was a response to someone else, there is nothing on my machine's logs that indicates any problem at all. The service simply fails to start, but when it runs it doesn't have any issues unless something configures an interface, leading to the DHCP service silent crash.

    @akuma1x said in DHCP fails silently, but works on reboot of pfSense:

    This is WAY off topic, so I apologize, and maybe it can be a separate topic.

    @obelsen - I'm very new to the use and tech, but why would there be a real-world use for so many VLAN's - you state 200 in your original post. How/when would anybody need to use that many virtual networks? I could see a handful being useful, but why a very large amount? I'm just curious...

    Jeff

    200 VLANs is hardly what I would expect as exceptional. We use it to ensure that all communication goes through our firewall, so we can filter traffic as we want per socket. Port isolation on our switches was another alternative, but gives less control.


  • Rebel Alliance Developer Netgate

    So there are no log messages at all in your case? Not even any interface events or other logs that seem to repeat like the other person seeing this?

    When you have dhcpd running, look at the output of ps uxaww | grep dhcpd and note the full command output. Next time it fails, try to run that by hand from an ssh or console shell and see if it produces any output. If it doesn't, try adding -d to the parameters before anything else, which should have it print output to the terminal.



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    So there are no log messages at all in your case? Not even any interface events or other logs that seem to repeat like the other person seeing this?

    None. In the DHCP log, only events related to new leases are present. When I try to restart the service, it only logs the listening/sending interfaces. Following every restart of the service, it's nothing until the same log spam of listening/sending interfaces, ending with the sending on socket/fallback.

    When you have dhcpd running, look at the output of ps uxaww | grep dhcpd and note the full command output. Next time it fails, try to run that by hand from an ssh or console shell and see if it produces any output. If it doesn't, try adding -d to the parameters before anything else, which should have it print output to the terminal.

    The DHCP server fails only when modifying interfaces. It does not crash when it is already running. I would expect this is because the daemon is restarted, and it is unable to start again.
    I will try to replicate the conditions again in a VM and run ps, however it did not enlighten me earlier when I did my own troubleshooting.


  • Rebel Alliance Developer Netgate

    @obelsen said in DHCP fails silently, but works on reboot of pfSense:

    None. In the DHCP log, only events related to new leases are present.

    What about in the main system log?

    @obelsen said in DHCP fails silently, but works on reboot of pfSense:

    The DHCP server fails only when modifying interfaces

    Do you have any special settings on that interface? Maybe a spoofed MAC address, MTU, or other similar setting either on the parent interface (if assigned) or one of the VLANs?



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    @obelsen said in DHCP fails silently, but works on reboot of pfSense:

    None. In the DHCP log, only events related to new leases are present.

    What about in the main system log?

    Negative. No logs at all. The only items present were standard login logs and other unrelated info.

    @obelsen said in DHCP fails silently, but works on reboot of pfSense:

    The DHCP server fails only when modifying interfaces

    Do you have any special settings on that interface? Maybe a spoofed MAC address, MTU, or other similar setting either on the parent interface (if assigned) or one of the VLANs?

    The interfaces are all standard settings with a static ip defined. No MAC spoofing or other non standard configuration.
    The parent interface (LAN) also had a static IP like each vlan interface.



  • I would like to add that this has been an issue for around a year, and I have previously written on the subreddit for pfSense for help to no avail.


  • Rebel Alliance Developer Netgate

    Can you answer the other questions I asked in my previous reply?



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    Can you answer the other questions I asked in my previous reply?

    I did, forgot to put a newline :)


  • Rebel Alliance Developer Netgate

    The symptoms all fit with something causing a link loop which generally only happens on certain drivers in certain situations such as changing specific settings which cause the link to drop and come back.

    That triggers the link up/down scripts, which reconfigure the interfaces, which triggers a new link event, and so on.

    But that scenario would log quite a lot of info in the main system log as it happens. It wouldn't happen silently.

    Are the affected NICs all igb interfaces?



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    The symptoms all fit with something causing a link loop which generally only happens on certain drivers in certain situations such as changing specific settings which cause the link to drop and come back.

    That triggers the link up/down scripts, which reconfigure the interfaces, which triggers a new link event, and so on.

    But that scenario would log quite a lot of info in the main system log as it happens. It wouldn't happen silently.

    Are the affected NICs all igb interfaces?

    It happens both when LAN is in the rj45 port (igb) and when moved to one of the spf+ ports (ix i believe)



  • Additionally, I forgot to mention that even restarting an openvpn server caused the problem.
    Openvpn does however not cause the problem itself, as an install without openvpn showed the same issues.


  • Rebel Alliance Developer Netgate

    Restarting OpenVPN would also trigger a restart of some services, which could land in a similar scenario depending on the circumstances.

    There must be something out of the ordinary on there that triggers it, however.

    What other packages are on there? Any other services on the firewall?

    I'd be interested in looking at a full copy of the config.xml if possible. You can redact some private info (passwords/certs/etc) but I'd like to see as much of it as possible. You can send it in a PM or send it to <my forum username>@pfsense.org and you can encrypt it with GPG/PGP if you like, there is a key on public key servers for that e-mail address.



  • I'll see what I can do, I'll take a look at it tomorrow and get back to you.



  • @jimp @obelsen
    Here is some additional information from my end. Rebooting the pfsense gets everything working fine again. Left things running for 2 days without changing anything and everything ran smoothly for that period of time. System Log was totally clean.
    The 2 days without changing anything was the time period it took to get a second X1541 shipped out to me. I couldn't keep rebooting to bring the DHCP server back on line every time I made an interface change.
    So with the second x1541 in place (I haven't set up HA yet), I am now making the interface changes on the off-line X1541 saving config and rebooting, then hot swapping to production. Then taking the now off-line X1541, restoring the config I saved with the updates. This is working for now to get me thru some immediate updates I need to make, but certainly not a long term solution.
    The clarify the syslog is totally clean until I make an interface change, then the log starts immediately filling up (as posted earlier)



  • @bjk said in DHCP fails silently, but works on reboot of pfSense:

    @jimp @obelsen
    Here is some additional information from my end. Rebooting the pfsense gets everything working fine again. Left things running for 2 days without changing anything and everything ran smoothly for that period of time. System Log was totally clean.
    The 2 days without changing anything was the time period it took to get a second X1541 shipped out to me. I couldn't keep rebooting to bring the DHCP server back on line every time I made an interface change.
    So with the second x1541 in place (I haven't set up HA yet), I am now making the interface changes on the off-line X1541 saving config and rebooting, then hot swapping to production. Then taking the now off-line X1541, restoring the config I saved with the updates. This is working for now to get me thru some immediate updates I need to make, but certainly not a long term solution.
    The clarify the syslog is totally clean until I make an interface change, then the log starts immediately filling up (as posted earlier)

    I don't think it's the exact same issue, as my system log does not have any indication of an issue when the DHCP server is down (other than that my service watchdog keeps restarting DHCPD)



  • It does seem like there is some overlap tho as we are both utilizing 100+ Vlans and the DHCP server stops functioning when any updates are made to Interfaces. Or maybe I'm over simplifying? In reading your posts, I want to say I'm seeing the same problem, only I do have system log activity. If our issues are somehow related, maybe the addition to me having the log activity is a clue? probably not... just throwing it out there.



  • @jimp As I am adding VLANs on the off-line x1541, I hit a snag. I was able to create the 163rd VLAN and select "Add" in the Interface Assignments, but now this latest VLAN I added isn't showing up in the Interface Assignments tab. If I go back to the VLANs tab, the VLAN I last created is there. When I try to delete the VLAN, I receive an error "This VLAN cannot be deleted because it is still being used as an interface". Yet it isn't showing up in the Interface list. I decided to reboot (after I saved the Config). Upon reboot, the last several VLANs were missing. I restored from the back up and was able to get back to where I was before the reboot (can see the VLAN but not the Interface).
    @obelsen, you didn't run into any troubles adding all your VLANs? Have I hit some limitation here?



  • @bjk said in DHCP fails silently, but works on reboot of pfSense:

    @jimp As I am adding VLANs on the off-line x1541, I hit a snag. I was able to create the 163rd VLAN and select "Add" in the Interface Assignments, but now this latest VLAN I added isn't showing up in the Interface Assignments tab. If I go back to the VLANs tab, the VLAN I last created is there. When I try to delete the VLAN, I receive an error "This VLAN cannot be deleted because it is still being used as an interface". Yet it isn't showing up in the Interface list. I decided to reboot (after I saved the Config). Upon reboot, the last several VLANs were missing. I restored from the back up and was able to get back to where I was before the reboot (can see the VLAN but not the Interface).
    @obelsen, you didn't run into any troubles adding all your VLANs? Have I hit some limitation here?

    I have not encountered this error.


  • Rebel Alliance Developer Netgate

    There is no limit on the number of VLANs you can have in the GUI. I'm not quite sure what would have happened there, but I recall seeing anything like that before. Are you certain there wasn't some unintentional overlap? IIRC input validation prevents most obvious ways of causing duplicate entries but maybe something else was going on there.

    Go to Diagnostics > Backup & Restore and check the config history at each point in the changes you made, see what was added when.



  • @jimp I've reviewed the interfaces a few times and cannot see any conflict or redundancies. Reviewing the XML, the VLAN and Interface are there, just doesn't show up in the UI.
    I also stepped backwards in the configs a couple times to re-add the vlan and also tried adding a completely random vlan with the same result of it not showing up after adding in the interfaces.
    From my experience, it appears I've hit a limit of not being able to add any more interfaces and have them show up in the UI... tho they are showing up in the XML



  • This post is deleted!

  • Rebel Alliance Developer Netgate

    So exactly how many VLAN tags do you have in config.xml when they stop showing up?

    And can you confirm which version of pfSense you are running on that device?



  • @jimp
    Hardware XG-1541 running 2.4.4-RELEASE (amd64)
    built on Thu Sep 20 09:33:19 EDT 2018
    FreeBSD 11.2-RELEASE-p3

    The GUI stopped displaying after the 163 VLAN was created. Also, please recall, I rebooted after to see if something magic would happen. And it did. The GUI was now not evern showing the 162 VLANs Ihad previously created. I didn't count, but all I could see in the GUI for interfaces was more like ~155 VLANs. I restored the config from before the reboot and then I could see the 162 VLANs again in the GUI. At this time, I was not reviewing the XML to see what it had in it. But something wonky for sure going on with the GUI not displaying all interfaces.

    btw, thank you very much for your help with this.


  • Rebel Alliance Developer Netgate

    I generated a config with 250 VLANs (assigned, enabled, with DHCP) and so far they all show up everywhere. The console menu, the interfaces widget, the interfaces menu, the VLAN tab list, the DHCP server, even firewall rules.

    I was able to reproduce some weirdness when applying a change to an interface, though. It's like it is very slowly going through each VLAN and touching it. PHP is maxing out a whole core, but it's only doing maybe one VLAN every 2 minutes. So at this rate it would take 8h20m to process them all. Fun. :-)

    So I can at least open an issue for that part, which may be the original cause of this thread. Not sure what is taking so long to process them all, or why it's touching every VLAN when changing only one. Might not make -p1 but we'll see.



  • @jimp said in DHCP fails silently, but works on reboot of pfSense:

    I generated a config with 250 VLANs (assigned, enabled, with DHCP) and so far they all show up everywhere. The console menu, the interfaces widget, the interfaces menu, the VLAN tab list, the DHCP server, even firewall rules.

    I was able to reproduce some weirdness when applying a change to an interface, though. It's like it is very slowly going through each VLAN and touching it. PHP is maxing out a whole core, but it's only doing maybe one VLAN every 2 minutes. So at this rate it would take 8h20m to process them all. Fun. :-)

    So I can at least open an issue for that part, which may be the original cause of this thread. Not sure what is taking so long to process them all, or why it's touching every VLAN when changing only one. Might not make -p1 but we'll see.

    Ah I can see that being the issue. I haven't actually tried leaving it running for that long, since people around here don't know how to set static IPs for that duration :-)


  • Rebel Alliance Developer Netgate

    I put in https://redmine.pfsense.org/issues/9115 for this for now. If I notice any other clues about how it misbehaves I'll drop them on there.

    In the meantime you might be able to at least do the 16/11 trick after making a change to keep it running. It does seem to apply the intended change quickly, and then gets mired in doing something else after.



  • This sounds like a good lead. thank you for going thru the effort. Should I start a new thread concerning why my VLAN interfaces are not showing up in the GUI? Any other thoughts on this? Any suggestions on things to try to maybe understand what might be happening here? thx again


  • Rebel Alliance Developer Netgate

    @bjk said in DHCP fails silently, but works on reboot of pfSense:

    Should I start a new thread concerning why my VLAN interfaces are not showing up in the GUI? Any other thoughts on this? Any suggestions on things to try to maybe understand what might be happening here? thx again

    Yeah that's a separate issue and I can't reproduce it here at all. I can't see why it would behave inconsistently either.